Oct 31 00:18:08.864865 ip-10-0-132-66 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:18:09.258989 ip-10-0-132-66 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:09.258989 ip-10-0-132-66 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:18:09.258989 ip-10-0-132-66 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:09.258989 ip-10-0-132-66 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:09.263379 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263298 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.263379 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263373 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.263379 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263378 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.263379 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263382 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.263379 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263386 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263391 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263395 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263399 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263402 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263406 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263410 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263413 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263417 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263421 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263425 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263430 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263436 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263440 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263444 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263448 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263453 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263457 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263461 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.263684 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263465 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263468 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263472 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263478 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263485 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263490 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263495 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263499 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263503 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263508 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263512 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263516 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263521 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263525 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263529 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263534 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263553 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263557 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263562 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.264460 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263567 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263571 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263576 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263580 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263584 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263589 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263593 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263598 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263602 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263606 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263610 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263615 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263619 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263623 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263627 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263632 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263636 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263641 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263646 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263651 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.265319 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263655 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263660 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263664 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263668 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263675 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263681 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263686 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263691 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263696 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263700 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263706 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263711 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263715 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263720 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263724 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263728 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263734 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263738 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263742 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263746 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.266168 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263750 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263755 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263759 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263764 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263769 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263773 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263777 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263782 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263786 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263790 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263795 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263799 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263804 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263809 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263813 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263817 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263821 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263826 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263830 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.267081 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263834 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263846 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263851 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.263856 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265146 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265156 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265161 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265166 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265171 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265175 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265179 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265183 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265188 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265191 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265196 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265200 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265206 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265210 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.267911 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265215 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265219 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265223 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265227 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265231 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265236 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265240 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265245 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265249 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265253 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265258 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265262 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265266 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265270 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265274 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265279 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265283 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265287 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265292 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265296 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.268749 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265301 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265305 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265309 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265313 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265317 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265321 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265325 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265329 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265332 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265336 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265344 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265351 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265356 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265361 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265365 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265370 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265374 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265379 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265383 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.269520 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265387 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265391 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265396 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265400 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265405 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265410 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265418 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265422 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265426 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265430 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265435 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265440 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265445 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265449 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265453 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265457 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265461 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265465 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265469 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.270036 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265473 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265478 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265482 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265487 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265491 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265496 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265500 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265504 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265508 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265515 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265522 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265527 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265532 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265555 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265560 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265564 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265568 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265572 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265577 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.270624 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265581 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265585 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265589 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265595 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265600 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265605 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265618 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265624 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265628 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265632 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265637 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265641 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265645 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.265649 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266464 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266474 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266478 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266483 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266488 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266493 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.271261 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266497 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266502 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266506 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266511 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266515 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266519 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266523 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266528 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266532 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266553 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266558 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266562 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266567 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266571 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266575 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266578 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266583 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266587 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266592 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266596 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.271787 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266601 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266605 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266609 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266614 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266618 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266622 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266626 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266631 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266635 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266640 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266644 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266648 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266653 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266657 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266661 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266666 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266671 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266675 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266679 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.272272 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266687 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266693 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266698 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266703 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266708 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266713 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266717 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266722 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266726 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266731 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266735 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266739 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266744 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266748 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266753 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266758 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266762 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266766 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266774 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.272827 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266781 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266786 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266791 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266796 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266801 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266805 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266811 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266815 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266820 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266825 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266829 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266834 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266839 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266845 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266850 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266854 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266859 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266863 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266867 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266871 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.273275 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266875 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266880 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266885 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266889 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266894 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266898 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266903 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266907 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266911 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266915 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266919 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266927 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266932 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266936 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266940 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266945 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266949 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266953 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266958 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.273777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266962 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266966 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.266970 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267102 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267113 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267125 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267131 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267138 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267143 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267150 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267156 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267162 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267167 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267172 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267177 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267182 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267187 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267192 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267197 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267202 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267207 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267213 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267218 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:18:09.274220 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267223 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267228 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267234 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267242 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267247 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267252 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267257 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267262 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267266 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267271 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267276 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267281 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267288 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267293 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267297 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267302 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267307 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267312 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267319 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267324 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267329 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267334 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267339 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267346 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:18:09.274769 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267350 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267355 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267360 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267366 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267370 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267375 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267380 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267385 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267389 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267395 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267406 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267413 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267418 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267423 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267428 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267432 2576 flags.go:64] FLAG: --help="false" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267437 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267442 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267447 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267452 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267457 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267463 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267468 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:18:09.275351 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267472 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267477 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267482 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267487 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267492 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267497 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267502 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267506 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267511 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267516 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267521 2576 flags.go:64] FLAG: --lock-file="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267526 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267530 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267552 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267561 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267566 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267571 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267576 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267581 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267586 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267592 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267597 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267604 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267609 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267615 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 00:18:09.275997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267620 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267625 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267630 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267635 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267639 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267644 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267649 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267660 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267664 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267669 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267674 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267679 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267684 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267688 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267693 2576 flags.go:64] FLAG: --port="10250" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267698 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267702 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06caefb75075e1bab" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267707 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267712 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267718 2576 flags.go:64] FLAG: --register-node="true" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267722 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267728 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267733 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267738 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267743 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267749 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:18:09.276617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267754 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267761 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267766 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267770 2576 flags.go:64] FLAG: --runonce="false" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267775 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267780 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267784 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267790 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267794 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267799 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267804 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267808 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267813 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267817 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267822 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267826 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267832 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267837 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267841 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267850 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267856 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267860 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267867 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267877 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267882 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:18:09.277273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267887 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267892 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267897 2576 flags.go:64] FLAG: --v="2" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267904 2576 flags.go:64] FLAG: --version="false" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267911 2576 flags.go:64] FLAG: --vmodule="" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267916 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.267922 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268139 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268148 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268154 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268159 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268164 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268168 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268173 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268177 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268181 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268186 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268190 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268194 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268199 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.277864 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268203 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268207 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268211 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268216 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268220 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268224 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268229 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268235 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268242 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268248 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268252 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268257 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268261 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268265 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268270 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268274 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268278 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268282 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268287 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.278338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268292 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268299 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268303 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268308 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268312 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268316 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268321 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268325 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268329 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268336 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268341 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268347 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268352 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268357 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268361 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268366 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268370 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268374 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268379 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268383 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.278823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268389 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268393 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268398 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268402 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268406 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268410 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268414 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268419 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268423 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268428 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268432 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268437 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268441 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268453 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268458 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268462 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268466 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268471 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268475 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.279287 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268479 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268483 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268487 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268492 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268496 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268501 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268505 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268509 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268513 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268518 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268522 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268526 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268532 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268551 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268556 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268560 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268564 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268568 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268572 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268577 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.279777 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268581 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268585 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268589 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268594 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268598 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268604 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268610 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268614 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268618 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268623 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268627 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268631 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268635 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268640 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.268682 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268737 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.280283 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268743 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268748 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268753 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268757 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268762 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268766 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268772 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268776 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268781 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268787 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268791 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268796 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268800 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268804 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268808 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268813 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268817 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268821 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268825 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268830 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.280682 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268835 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268841 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268845 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268851 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268855 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268859 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268863 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268867 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268872 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268876 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268881 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268885 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268889 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268894 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268898 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268902 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268907 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268911 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268915 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268920 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.281141 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268924 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268930 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268934 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268938 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268942 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268947 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268951 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268955 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268959 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268964 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268968 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268972 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268977 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268983 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268987 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268991 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.268996 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269001 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269005 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.281616 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269010 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269014 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269019 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269023 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269027 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269032 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269036 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269041 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269045 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269049 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269054 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269058 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269062 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269067 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269072 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269077 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269081 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269085 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269089 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.282062 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269094 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269098 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269102 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269107 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269111 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269117 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269123 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269132 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269137 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269142 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269146 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269151 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269155 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269159 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269164 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269168 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269173 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269178 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269182 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.282504 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269186 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269191 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269195 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269199 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269203 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269207 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269215 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.269221 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.277963 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.277978 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.278777 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.278788 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278853 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278858 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278861 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278864 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278867 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278870 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278872 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.282978 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278875 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278878 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278881 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278883 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278886 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278888 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278891 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278894 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278896 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278899 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278901 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278904 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278907 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278909 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278912 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278914 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278917 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278920 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278923 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278925 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.283425 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278928 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278931 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278939 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278942 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278945 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278947 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278950 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278953 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278955 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278958 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278961 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278963 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278966 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278968 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278971 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278974 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278976 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278979 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278981 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.283931 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278984 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278986 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278989 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278992 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278994 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278997 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.278999 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279002 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279005 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279007 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279010 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279013 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279015 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279018 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279021 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279023 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279035 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279038 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279041 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.284366 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279044 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279052 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279055 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279058 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279061 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279063 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279066 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279068 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279072 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279076 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279079 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279082 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279084 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279087 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279090 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279092 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279095 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279098 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279100 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.284823 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279103 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279106 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279109 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279111 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279114 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279117 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279119 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279122 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279125 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279130 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279139 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279142 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279146 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279150 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279153 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279156 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279159 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279161 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.285298 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279164 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279167 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279170 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.279198 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279424 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279429 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279432 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279436 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279438 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279441 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279444 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279446 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279452 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279454 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279457 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.285760 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279459 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279462 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279465 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279468 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279471 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279474 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279477 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279479 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279482 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279491 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279494 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279497 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279499 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279502 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279504 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279508 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279511 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279515 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279518 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.286351 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279520 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279523 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279526 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279528 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279531 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279533 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279552 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279557 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279561 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279564 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279568 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279571 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279573 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279576 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279578 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279581 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279584 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279591 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279594 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.287092 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279597 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279600 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279603 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279605 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279614 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279617 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279619 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279622 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279624 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279627 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279629 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279632 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279634 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279637 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279640 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279642 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279645 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279647 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279650 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279652 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.288075 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279655 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279657 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279660 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279662 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279665 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279667 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279670 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279672 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279675 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279678 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279680 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279683 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279685 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279688 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279690 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279693 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279695 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279704 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279707 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.288752 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279709 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279712 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279714 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279717 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279719 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279722 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279724 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279726 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279729 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279731 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279734 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279736 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279739 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279741 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279744 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279746 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.289212 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279749 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.279772 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279812 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279816 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279820 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279823 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279825 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279828 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279831 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279834 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279836 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279839 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279842 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279844 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279847 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279855 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:09.289625 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279857 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279860 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279862 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279865 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279867 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279876 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279879 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279882 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279884 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279887 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279889 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279892 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279894 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279897 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279899 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279902 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279905 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279907 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279910 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279912 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:09.289998 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279915 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279917 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279920 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279922 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279925 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279928 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279930 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279933 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279935 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279938 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279940 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279943 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279954 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279957 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279960 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279962 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279965 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279967 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279970 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279972 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:09.290517 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279976 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279978 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279980 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279983 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279985 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279988 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279990 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279993 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279995 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.279998 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280001 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280003 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280006 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280008 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280011 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280013 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280016 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280018 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280021 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280023 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:09.291338 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280026 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280036 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280039 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280041 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280049 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280052 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280055 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280057 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280060 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280063 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280065 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280069 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280071 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280074 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280076 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280079 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280081 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280083 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280086 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:09.291860 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280090 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280093 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280096 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280099 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280101 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280104 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280107 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280109 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280112 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280114 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280117 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:09.280122 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.280179 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.282613 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.283677 2576 server.go:1014] "Starting client certificate rotation" Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.283772 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.283804 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:18:09.292360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.287701 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:09.292904 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.291392 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:09.301393 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.301379 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:18:09.305381 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.305367 2576 log.go:25] "Validated CRI v1 image API" Oct 31 00:18:09.306214 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.306193 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:18:09.309496 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.309476 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e306abcc-6c4a-478a-a64a-9cbc6ffe5afe:/dev/nvme0n1p3 fc862f49-e223-411d-9db6-7023170324ea:/dev/nvme0n1p4] Oct 31 00:18:09.309588 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.309495 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:18:09.315764 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.315662 2576 manager.go:217] Machine: {Timestamp:2026-10-31 00:18:09.31414257 +0000 UTC m=+0.339836554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100195 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f02a0e181c03a3deaa164a9db614 SystemUUID:ec22f02a-0e18-1c03-a3de-aa164a9db614 BootID:2f193b13-9f07-4fdf-b63c-98e3da3391fe Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:f3:29:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:f3:29:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:f4:b1:cb:72:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:18:09.315764 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.315763 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:18:09.315875 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.315856 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:18:09.320365 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320341 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:18:09.320491 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320366 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:18:09.320531 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320494 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:18:09.320531 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320502 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:18:09.320592 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320533 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:09.320592 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.320556 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:09.321813 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.321803 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:18:09.326294 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.326282 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:18:09.326410 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.326400 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:18:09.327258 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327248 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:18:09.327312 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327271 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:18:09.327312 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327285 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:18:09.327312 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327293 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:18:09.327312 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327301 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:18:09.327610 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.327598 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:09.330309 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.330288 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:18:09.338005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.337987 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:09.338959 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.338945 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:18:09.339012 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.338977 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:18:09.339131 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339121 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:18:09.339167 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339136 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:18:09.339167 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339142 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:18:09.339167 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339147 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:18:09.339167 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339153 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:18:09.339167 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339158 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:18:09.339914 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339905 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:18:09.339944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339915 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:18:09.339944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339922 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:18:09.339944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339928 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:18:09.339944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339937 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:18:09.339944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.339945 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:18:09.341402 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.341390 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:18:09.341439 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.341404 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:18:09.345267 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.345254 2576 server.go:1263] "Started kubelet" Oct 31 00:18:09.345370 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.345334 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:18:09.345427 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.345343 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:18:09.345495 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.345449 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:18:09.346066 ip-10-0-132-66 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:18:09.347124 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.346939 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:18:09.349511 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.349492 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:18:09.349923 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.349910 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:18:09.355173 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.355158 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:18:09.355781 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.355767 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:18:09.356054 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.356038 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.356684 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.356667 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:18:09.356684 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.356669 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:18:09.356684 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.356689 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:18:09.356810 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.356776 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:18:09.356810 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.356787 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:18:09.357201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357185 2576 factory.go:55] Registering systemd factory Oct 31 00:18:09.357201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357202 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:18:09.357387 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357376 2576 factory.go:153] Registering CRI-O factory Oct 31 00:18:09.357387 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357388 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 00:18:09.357462 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357430 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:18:09.357462 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357444 2576 factory.go:103] Registering Raw factory Oct 31 00:18:09.357462 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.357456 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 00:18:09.358472 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.358461 2576 manager.go:319] Starting recovery of all containers Oct 31 00:18:09.369804 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.369783 2576 manager.go:324] Recovery completed Oct 31 00:18:09.371974 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.371743 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 00:18:09.374492 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.374477 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.376636 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.376607 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.376700 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.376652 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.376700 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.376666 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.377243 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377229 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:18:09.377243 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377240 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:18:09.377348 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377259 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.377348 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377259 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:18:09.377972 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377951 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.377972 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377982 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.378107 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.377998 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.378157 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.378105 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:18:09.379459 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.379435 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:18:09.379553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.379504 2576 policy_none.go:50] "Start" Oct 31 00:18:09.379553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.379518 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:18:09.379553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.379531 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:18:09.380586 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.379441 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-66.ec2.internal.18e375278c841c70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-66.ec2.internal,},FirstTimestamp:2026-10-31 00:18:09.345231984 +0000 UTC m=+0.370925968,LastTimestamp:2026-10-31 00:18:09.345231984 +0000 UTC m=+0.370925968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-66.ec2.internal,}" Oct 31 00:18:09.381490 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.381479 2576 policy_none.go:44] "Start" Oct 31 00:18:09.390787 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.390768 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:18:09.394562 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.394522 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:18:09.394814 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.394792 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 00:18:09.394914 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.394819 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-66.ec2.internal.18e375278e634b62 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-66.ec2.internal,},FirstTimestamp:2026-10-31 00:18:09.376635746 +0000 UTC m=+0.402329739,LastTimestamp:2026-10-31 00:18:09.376635746 +0000 UTC m=+0.402329739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-66.ec2.internal,}" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.411899 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-66.ec2.internal.18e375278e63a7a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-66.ec2.internal,},FirstTimestamp:2026-10-31 00:18:09.376659364 +0000 UTC m=+0.402353350,LastTimestamp:2026-10-31 00:18:09.376659364 +0000 UTC m=+0.402353350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-66.ec2.internal,}" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.412341 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.419942 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsw64" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.421165 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.422318 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.422339 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.422355 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.422384 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:18:09.430347 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.427010 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:18:09.433043 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.431891 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsw64" Oct 31 00:18:09.434098 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.434031 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-66.ec2.internal.18e375278e63d670 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-66.ec2.internal,},FirstTimestamp:2026-10-31 00:18:09.376671344 +0000 UTC m=+0.402365330,LastTimestamp:2026-10-31 00:18:09.376671344 +0000 UTC m=+0.402365330,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-66.ec2.internal,}" Oct 31 00:18:09.442851 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.442836 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 00:18:09.442941 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.442864 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:18:09.442941 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.442875 2576 server.go:87] "Starting device plugin registration server" Oct 31 00:18:09.443112 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.443099 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:18:09.443174 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.443113 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:18:09.443228 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.443179 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:18:09.443300 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.443250 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:18:09.443300 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.443258 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:18:09.443903 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.443883 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:18:09.443973 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.443936 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.451144 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.451054 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-66.ec2.internal.18e375278e634b62\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-66.ec2.internal.18e375278e634b62 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-66.ec2.internal,},FirstTimestamp:2026-10-31 00:18:09.376635746 +0000 UTC m=+0.402329739,LastTimestamp:2026-10-31 00:18:09.377971274 +0000 UTC m=+0.403665265,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-66.ec2.internal,}" Oct 31 00:18:09.481393 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.481371 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:09.522760 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.522719 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal"] Oct 31 00:18:09.522824 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.522769 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.523533 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.523518 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.523621 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.523560 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.523621 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.523572 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.525835 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.525823 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.525972 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.525958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.526006 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.525985 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.526475 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526458 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.526560 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526458 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.526560 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526489 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.526560 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526503 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.526560 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526502 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.526683 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.526577 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.528635 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.528622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.528686 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.528643 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.529262 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.529248 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.529326 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.529272 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.529326 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.529281 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.543463 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.543342 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:09.544047 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.544033 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:09.544122 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.544058 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:09.544122 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.544071 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:09.544122 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.544100 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.557602 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.557582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.557677 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.557606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.557677 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.557623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.561187 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.561172 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-66.ec2.internal\" not found" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.563584 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.563560 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-66.ec2.internal\" not found" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.565354 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.565340 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.565433 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.565362 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-66.ec2.internal\": node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.598986 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.598969 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.657902 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.657997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.657997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.657997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.657997 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.658137 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.657988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.699977 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.699959 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.800328 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.800279 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:09.865974 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.865955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.869109 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:09.869093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:09.900743 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:09.900721 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:10.001192 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:10.001167 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:10.101713 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:10.101666 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.452782 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.452752 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.534274 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.534250 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:18:11.534649 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.534374 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:18:11.553515 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.553497 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.653886 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.653829 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.665661 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.665642 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:11.666830 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.666809 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:11.666908 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.666840 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:11.666908 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.666850 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:11.688243 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.688201 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:10 +0000 UTC" deadline="2028-03-30 04:47:27.42503668 +0000 UTC" Oct 31 00:18:11.688358 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.688244 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12388h29m15.736797771s" Oct 31 00:18:11.716934 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:11.716905 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4683e0633b522c7ed8c0620830924e81.slice/crio-e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a WatchSource:0}: Error finding container e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a: Status 404 returned error can't find the container with id e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a Oct 31 00:18:11.717601 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:11.717580 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77ff0c82f82c3043aa6a2e6855afd44.slice/crio-6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379 WatchSource:0}: Error finding container 6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379: Status 404 returned error can't find the container with id 6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379 Oct 31 00:18:11.721702 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.721689 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:18:11.726758 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.726739 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:11.754149 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.754128 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.776427 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:11.776406 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:11.854805 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.854777 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:11.955329 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:11.955276 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:12.055929 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.055908 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:12.131559 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.131533 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:12.207785 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.207586 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:12.235921 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.235899 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:12.237079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.237055 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:12.257682 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.257663 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:12.513107 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.513038 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:12.579438 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.579415 2576 apiserver.go:52] "Watching apiserver" Oct 31 00:18:12.607196 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.607168 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:18:12.626305 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.626283 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:18:12.626662 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.626634 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal","openshift-multus/multus-m67hq","openshift-multus/network-metrics-daemon-7mtxv","openshift-network-diagnostics/network-check-target-q2pbw","kube-system/konnectivity-agent-klndj","openshift-cluster-node-tuning-operator/tuned-gjzn8","openshift-multus/multus-additional-cni-plugins-rnzgf","openshift-dns/node-resolver-j9cp4","openshift-network-operator/iptables-alerter-b95dr","openshift-ovn-kubernetes/ovnkube-node-n26th","kube-system/global-pull-secret-syncer-77tnr","kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c","openshift-image-registry/node-ca-krnph"] Oct 31 00:18:12.629358 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.629332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.631496 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.631477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.634104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.633739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.636123 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.635930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:12.636123 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.636000 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:12.638360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.638177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.640489 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.640472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.643232 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.643082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.643232 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.643182 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:12.645556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.645538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.647503 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647485 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.647599 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647584 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:18:12.647663 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647650 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:18:12.647765 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647751 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\"" Oct 31 00:18:12.647832 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647817 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.647923 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647891 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:18:12.647990 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.647977 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\"" Oct 31 00:18:12.648420 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.648384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.651352 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.651331 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.654443 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.654426 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:18:12.654889 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.654867 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.655867 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.655847 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:18:12.657930 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.657906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.658027 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.657973 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:12.660331 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.660311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.660648 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.660631 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:18:12.660741 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.660680 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wtkxk" Oct 31 00:18:12.662720 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.662702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.663918 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.663902 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wtkxk" Oct 31 00:18:12.678288 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.678245 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerStarted","Data":"6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379"} Oct 31 00:18:12.679606 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.679581 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" event={"ID":"4683e0633b522c7ed8c0620830924e81","Type":"ContainerStarted","Data":"e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a"} Oct 31 00:18:12.709660 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.708608 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:18:12.717756 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.717851 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.717851 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgpn\" (UniqueName: \"kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.717851 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.718005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.718005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.718005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99nn\" (UniqueName: \"kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.718005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.717979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718221 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.718556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkgh\" (UniqueName: \"kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnrk\" (UniqueName: \"kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.719098 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.718974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nttdf\" (UniqueName: \"kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcq4h\" (UniqueName: \"kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82nk\" (UniqueName: \"kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpwk\" (UniqueName: \"kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.719847 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8qj\" (UniqueName: \"kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.719997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.720517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.721136 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.721136 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.721136 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.721136 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.721136 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.720355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.747767 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.747744 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.748259 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748114 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.748259 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748131 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.748416 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748266 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:18:12.748416 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748298 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.748416 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748331 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.748938 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748914 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.749023 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748922 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:18:12.749023 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.748984 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:18:12.749023 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749019 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\"" Oct 31 00:18:12.749156 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749022 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.749156 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749042 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:18:12.749156 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749123 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:18:12.749156 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749147 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749177 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749221 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749234 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749262 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749304 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\"" Oct 31 00:18:12.749325 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749315 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\"" Oct 31 00:18:12.749564 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749352 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\"" Oct 31 00:18:12.749564 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749263 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:18:12.749564 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.749396 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:18:12.762496 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.762459 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\"" Oct 31 00:18:12.762496 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.762477 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:18:12.821507 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.821637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.821637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.821637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.821637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.821637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkgh\" (UniqueName: \"kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnrk\" (UniqueName: \"kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.821834 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttdf\" (UniqueName: \"kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.822248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.821938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822470 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.822518 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822518 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822603 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822603 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822603 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcq4h\" (UniqueName: \"kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.822710 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82nk\" (UniqueName: \"kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.822710 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822710 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpwk\" (UniqueName: \"kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.822710 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.822880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.823155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.823155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.822937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.823367 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.823482 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823482 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8qj\" (UniqueName: \"kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823482 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.823625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.823625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.823797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.823797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.823797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.823797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.823797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.823956 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.823956 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.823956 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgpn\" (UniqueName: \"kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.823956 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.823956 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.824104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.824104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.823995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99nn\" (UniqueName: \"kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.824104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.824104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824104 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.824318 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.824553 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.824831 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.824831 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.824990 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.824965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.825045 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.825031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.825099 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.825085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.825146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.825132 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.825501 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.825453 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:18:12.829366 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.829346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.829482 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.829461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.829590 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.829525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.830155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.830273 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.830365 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.830476 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.830639 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.830763 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.830838 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830824 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.830908 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.830892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.831135 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831113 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.831251 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831571 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.831984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832015 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832481 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832615 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832811 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.832981 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833003 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833369 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833581 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833836 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.834946 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.833932 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.834037 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.834117 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:13.334082294 +0000 UTC m=+3.109108135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.834274 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.834377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:13.334358467 +0000 UTC m=+3.109384307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:12.835554 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.834681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.836114 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.835672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.836114 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.835959 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.836114 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.836106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.847925 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.847870 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkgh\" (UniqueName: \"kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.849570 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.849544 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8qj\" (UniqueName: \"kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:12.856407 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.856390 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgpn\" (UniqueName: \"kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:12.859187 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.859164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnrk\" (UniqueName: \"kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.862075 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.862054 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:12.862171 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.862079 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:12.862171 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.862092 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:12.862171 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:12.862147 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:13.362130572 +0000 UTC m=+3.137156413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:12.864137 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.864107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpwk\" (UniqueName: \"kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:12.864603 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.864586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99nn\" (UniqueName: \"kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.868951 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.868906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82nk\" (UniqueName: \"kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.870490 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.870469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcq4h\" (UniqueName: \"kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:12.874059 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.874028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttdf\" (UniqueName: \"kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:12.962371 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.962343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:12.969637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.969613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:12.971601 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:12.971573 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171f86cc_abaf_4fec_8ca2_d4809d731a37.slice/crio-bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56 WatchSource:0}: Error finding container bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56: Status 404 returned error can't find the container with id bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56 Oct 31 00:18:12.975045 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.975025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:12.976174 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.976156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:12.978909 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:12.978826 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf5110e_cdaf_4b6c_9d82_ceefd5b52a25.slice/crio-087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab WatchSource:0}: Error finding container 087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab: Status 404 returned error can't find the container with id 087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab Oct 31 00:18:12.982286 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.982269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:12.988924 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:12.988898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m67hq" Oct 31 00:18:13.011182 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.011161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:13.013479 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.013460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:13.013891 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:13.013872 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bfbf31_81b8_4b9b_87ef_86f5c1ad5fa1.slice/crio-666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314 WatchSource:0}: Error finding container 666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314: Status 404 returned error can't find the container with id 666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314 Oct 31 00:18:13.014919 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.014875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:13.016666 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:13.016579 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b345b2_4e5d_4510_8d14_17dacce01a56.slice/crio-0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288 WatchSource:0}: Error finding container 0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288: Status 404 returned error can't find the container with id 0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288 Oct 31 00:18:13.017920 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:13.017896 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5574e12_afad_4304_b163_f30046d15fc1.slice/crio-8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9 WatchSource:0}: Error finding container 8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9: Status 404 returned error can't find the container with id 8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9 Oct 31 00:18:13.027675 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:13.027653 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5adc70d_63ec_4ab8_9ca5_8fb4928040fc.slice/crio-fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca WatchSource:0}: Error finding container fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca: Status 404 returned error can't find the container with id fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca Oct 31 00:18:13.029715 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:13.029690 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc26e1_5d51_46e9_8832_d12588bc7ac3.slice/crio-49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a WatchSource:0}: Error finding container 49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a: Status 404 returned error can't find the container with id 49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a Oct 31 00:18:13.432298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.430665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:13.432298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.430876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:13.432298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.430918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:13.432592 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.432464 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:13.432592 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.432540 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:14.432520398 +0000 UTC m=+4.207546234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:13.433168 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433150 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:13.433276 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433172 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:13.433276 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433185 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:13.433276 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433251 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:14.433235685 +0000 UTC m=+4.208261520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:13.433447 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433367 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:13.433447 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:13.433416 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:14.433404146 +0000 UTC m=+4.208429982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:13.667341 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.667266 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:12 +0000 UTC" deadline="2028-08-08 18:28:02.733708616 +0000 UTC" Oct 31 00:18:13.667341 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.667330 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15546h9m49.066383488s" Oct 31 00:18:13.697792 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.697067 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" event={"ID":"4683e0633b522c7ed8c0620830924e81","Type":"ContainerStarted","Data":"d8f364ee657e518ccd3a73e0c588baa34a3b5e8cfadb5911aa7f16052aade9f3"} Oct 31 00:18:13.701615 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.701515 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a"} Oct 31 00:18:13.707157 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.707089 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" event={"ID":"29dcb073-2ad3-4d21-91e2-24e2536de375","Type":"ContainerStarted","Data":"7a9532d2854826e9e21115547d112a50379b0bc9f7cc9e30a88ac3223fc4c879"} Oct 31 00:18:13.709540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.709475 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m67hq" event={"ID":"d5574e12-afad-4304-b163-f30046d15fc1","Type":"ContainerStarted","Data":"8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9"} Oct 31 00:18:13.713286 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.713186 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" event={"ID":"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25","Type":"ContainerStarted","Data":"087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab"} Oct 31 00:18:13.721527 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.720974 2576 generic.go:356] "Generic (PLEG): container finished" podID="e77ff0c82f82c3043aa6a2e6855afd44" containerID="3451fb5be0ad0da16d40db3a2802c1deb0ea19d82d4d30fbc8e511ce1599f065" exitCode=0 Oct 31 00:18:13.721527 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.721042 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerDied","Data":"3451fb5be0ad0da16d40db3a2802c1deb0ea19d82d4d30fbc8e511ce1599f065"} Oct 31 00:18:13.730723 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.730651 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9cp4" event={"ID":"e127939f-dcd5-405f-97c6-87b7a68a5c36","Type":"ContainerStarted","Data":"289dd2627806f2e7f1ecb589309cf1a577c70c0d7a716941d92bca680f8400da"} Oct 31 00:18:13.744059 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.744032 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-krnph" event={"ID":"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc","Type":"ContainerStarted","Data":"fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca"} Oct 31 00:18:13.746701 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.746677 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-klndj" event={"ID":"45b345b2-4e5d-4510-8d14-17dacce01a56","Type":"ContainerStarted","Data":"0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288"} Oct 31 00:18:13.765824 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.765799 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b95dr" event={"ID":"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1","Type":"ContainerStarted","Data":"666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314"} Oct 31 00:18:13.774637 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.774612 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerStarted","Data":"bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56"} Oct 31 00:18:13.786409 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:13.786366 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" podStartSLOduration=1.78635478 podStartE2EDuration="1.78635478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:13.744197831 +0000 UTC m=+3.519223675" watchObservedRunningTime="2026-10-31 00:18:13.78635478 +0000 UTC m=+3.561380717" Oct 31 00:18:14.439770 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.439732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:14.439965 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.439794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:14.439965 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.439824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:14.439965 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.439952 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:14.440110 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440010 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:16.439994084 +0000 UTC m=+6.215019911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:14.440436 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440419 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:14.440516 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440446 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:14.440516 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440459 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:14.440516 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440506 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:16.440491226 +0000 UTC m=+6.215517052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:14.440662 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440564 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:14.440662 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.440599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:16.440587954 +0000 UTC m=+6.215613776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:14.667778 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.667729 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:12 +0000 UTC" deadline="2028-07-22 19:06:11.814907248 +0000 UTC" Oct 31 00:18:14.667778 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.667776 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15138h47m57.147135644s" Oct 31 00:18:14.674083 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.674061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:14.674195 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.674158 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:14.674471 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.674457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:14.674556 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.674521 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:14.674615 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.674570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:14.674666 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:14.674614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:14.798944 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.798849 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerStarted","Data":"733b339927e41432fc292de322f26cf6b7ddf3edb344f8e614a884bc47f4484f"} Oct 31 00:18:14.844457 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:14.844400 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" podStartSLOduration=2.844361244 podStartE2EDuration="2.844361244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:14.843610506 +0000 UTC m=+4.618636365" watchObservedRunningTime="2026-10-31 00:18:14.844361244 +0000 UTC m=+4.619387104" Oct 31 00:18:15.806797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:15.806722 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="faee24d0f8163fca2d9dd89b53547699d5d8f2552fa597e86351f3efdec5a360" exitCode=0 Oct 31 00:18:15.807878 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:15.807854 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"faee24d0f8163fca2d9dd89b53547699d5d8f2552fa597e86351f3efdec5a360"} Oct 31 00:18:16.457973 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.457915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:16.458153 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.457978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:16.458153 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.458006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:16.458296 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458165 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:16.458296 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458245 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:20.458225445 +0000 UTC m=+10.233251280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:16.458726 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458704 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:16.458831 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458732 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:16.458831 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458744 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:16.458831 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:20.458777079 +0000 UTC m=+10.233802914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:16.458982 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458856 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:16.458982 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.458889 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:20.458878557 +0000 UTC m=+10.233904393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:16.675009 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.674404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:16.675009 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.674552 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:16.675009 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.674913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:16.675593 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.675398 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:16.675593 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:16.675477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:16.675593 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:16.675556 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:17.817418 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:17.817375 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-klndj" event={"ID":"45b345b2-4e5d-4510-8d14-17dacce01a56","Type":"ContainerStarted","Data":"8f35c0393407d032ef87edd6b321ed9fa48d7a9a81480ee8590353cc1e3ba87d"} Oct 31 00:18:17.859940 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:17.859861 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-klndj" podStartSLOduration=3.241006989 podStartE2EDuration="6.859843604s" totalImagesPullingTime="3.618836615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:17.858858831 +0000 UTC m=+7.633884675" watchObservedRunningTime="2026-10-31 00:18:17.859843604 +0000 UTC m=+7.634869448" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:18.676015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:18.676154 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:18.676516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:18.676596 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:18.676643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:18.677003 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:18.676709 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:18.820652 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:18.820624 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-krnph" event={"ID":"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc","Type":"ContainerStarted","Data":"470c0d1dc3d3c455f6d920073c6949f680dcbb82c42ea76f6eb558ef7a1c7426"} Oct 31 00:18:18.823457 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:18.823434 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" event={"ID":"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25","Type":"ContainerStarted","Data":"0ca966bf751de904d8547bbadc5d5c002e3cb50c08288dec30d1d34c4ee3b216"} Oct 31 00:18:20.012101 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.012035 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:18:20.491322 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.491294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:20.491443 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.491361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:20.491511 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.491452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:20.491632 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.491610 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:20.491731 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.491678 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.491658704 +0000 UTC m=+18.266684546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:20.492097 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492075 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:20.492195 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492101 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:20.492195 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492114 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:20.492195 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492159 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.492143466 +0000 UTC m=+18.267169287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:20.492378 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492236 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:20.492378 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.492270 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:28.492259462 +0000 UTC m=+18.267285307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:20.674770 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.674705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:20.674896 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.674834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:20.675193 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.675174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:20.675314 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.675268 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:20.675412 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.675394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:20.675523 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:20.675459 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:20.698588 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.698492 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:18:20.012055486Z","UUID":"0788ea48-23ae-4c17-8ed8-09c2d6ac7ce4","Handler":null,"Name":"","Endpoint":""} Oct 31 00:18:20.699953 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.699928 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:18:20.699953 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.699952 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:18:20.828917 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.828882 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m67hq" event={"ID":"d5574e12-afad-4304-b163-f30046d15fc1","Type":"ContainerStarted","Data":"121e06fb6ed7403b98bf7f33fefe95aa7170a3693e41957a5e9a93c1eb7bf115"} Oct 31 00:18:20.831773 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.831744 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" event={"ID":"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25","Type":"ContainerStarted","Data":"74e404465ef2701cbc1d6bcc6005d73b973ffbb16b3002adebc07d7beda879e7"} Oct 31 00:18:20.834001 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.833971 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="6d36e80c1d5c1aed701c527ee9a38453f5d1100fc01b2233249d8dc8f22c5aea" exitCode=0 Oct 31 00:18:20.834091 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.834016 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"6d36e80c1d5c1aed701c527ee9a38453f5d1100fc01b2233249d8dc8f22c5aea"} Oct 31 00:18:20.872785 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.872710 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-krnph" podStartSLOduration=5.217215508 podStartE2EDuration="9.87269432s" totalImagesPullingTime="4.655478812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:18.862458304 +0000 UTC m=+8.637484147" watchObservedRunningTime="2026-10-31 00:18:20.87269432 +0000 UTC m=+10.647720166" Oct 31 00:18:20.918884 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:20.918844 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-m67hq" podStartSLOduration=2.554274228 podStartE2EDuration="9.918831457s" totalImagesPullingTime="7.364557229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:20.87368162 +0000 UTC m=+10.648707465" watchObservedRunningTime="2026-10-31 00:18:20.918831457 +0000 UTC m=+10.693857300" Oct 31 00:18:21.839161 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.838962 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" event={"ID":"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25","Type":"ContainerStarted","Data":"f1761a273847f837b01233c806b048d7e27265c1cfdd56d502afc3f9b5fa65a4"} Oct 31 00:18:21.840874 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.840840 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9cp4" event={"ID":"e127939f-dcd5-405f-97c6-87b7a68a5c36","Type":"ContainerStarted","Data":"a37da38fda65f8cbada9e555f80035316edfe376104c8b29401830bd1dfddfe0"} Oct 31 00:18:21.843729 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.843707 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="d4adb56a84feb7f0d489006fbc3d90f4640874125159b87cb15f54bb971c7487" exitCode=0 Oct 31 00:18:21.843853 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.843810 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"d4adb56a84feb7f0d489006fbc3d90f4640874125159b87cb15f54bb971c7487"} Oct 31 00:18:21.882703 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.882656 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" podStartSLOduration=4.237524371 podStartE2EDuration="11.882643886s" totalImagesPullingTime="7.645119515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:21.881973498 +0000 UTC m=+11.657013512" watchObservedRunningTime="2026-10-31 00:18:21.882643886 +0000 UTC m=+11.657669729" Oct 31 00:18:21.979574 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:21.979520 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j9cp4" podStartSLOduration=3.214294298 podStartE2EDuration="10.979502644s" totalImagesPullingTime="7.765208346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:21.97912746 +0000 UTC m=+11.754153304" watchObservedRunningTime="2026-10-31 00:18:21.979502644 +0000 UTC m=+11.754528489" Oct 31 00:18:22.661784 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.661721 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:22.662240 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.662202 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:22.673557 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.673536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:22.673688 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.673621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:22.673688 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:22.673629 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:22.673688 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.673651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:22.673852 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:22.673719 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:22.673852 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:22.673780 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:22.848515 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.848477 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b95dr" event={"ID":"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1","Type":"ContainerStarted","Data":"512cb2433af3d51587425f60e056ff3791ab4801a92d3154c39fb5499cd70c6e"} Oct 31 00:18:22.889845 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:22.889804 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b95dr" podStartSLOduration=5.181778017 podStartE2EDuration="12.889791491s" totalImagesPullingTime="7.708013474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:22.889668573 +0000 UTC m=+12.664694434" watchObservedRunningTime="2026-10-31 00:18:22.889791491 +0000 UTC m=+12.664817334" Oct 31 00:18:23.852717 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:23.852669 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" event={"ID":"29dcb073-2ad3-4d21-91e2-24e2536de375","Type":"ContainerStarted","Data":"2c4566dfeee4d9446e8e93102acd433c6897d0e2caa8450ae1029291ba5a7028"} Oct 31 00:18:23.855528 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:23.855502 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="334b3e266b8d71b1a1e0d60eedf99c1e0b16e71a5766eace8d608bce4b2f5b8f" exitCode=0 Oct 31 00:18:23.855671 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:23.855590 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"334b3e266b8d71b1a1e0d60eedf99c1e0b16e71a5766eace8d608bce4b2f5b8f"} Oct 31 00:18:23.896499 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:23.896457 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" podStartSLOduration=2.923813068 podStartE2EDuration="12.896444085s" totalImagesPullingTime="9.972631017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:23.895879241 +0000 UTC m=+13.670905087" watchObservedRunningTime="2026-10-31 00:18:23.896444085 +0000 UTC m=+13.671469957" Oct 31 00:18:24.673531 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:24.673498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:24.673531 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:24.673540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:24.673731 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:24.673652 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:24.673965 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:24.673940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:24.674051 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:24.674035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:24.674124 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:24.674104 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:25.862855 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:25.862677 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="edc1b52528d1aceb89a463291618d547a736559a6b7afe2fbda325b8fce0b629" exitCode=0 Oct 31 00:18:25.863303 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:25.862778 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"edc1b52528d1aceb89a463291618d547a736559a6b7afe2fbda325b8fce0b629"} Oct 31 00:18:26.674285 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:26.674246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:26.674472 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:26.674375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:26.674472 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:26.674384 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:26.674597 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:26.674488 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:26.674597 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:26.674554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:26.674683 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:26.674632 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:26.868396 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:26.868364 2576 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="25da4a3578ba1217e4381a7bacfef2558435a8ab1a9e250308dde791855f9a4f" exitCode=0 Oct 31 00:18:26.868753 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:26.868407 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerDied","Data":"25da4a3578ba1217e4381a7bacfef2558435a8ab1a9e250308dde791855f9a4f"} Oct 31 00:18:27.098680 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.098654 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:27.098808 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.098759 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:27.099365 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.099346 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:27.873060 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.872893 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"125965d60711fa5f77daab8b71aa1f7b80ddcf8cba2ec574584201a4d6ef7d83"} Oct 31 00:18:27.873478 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.873105 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"9983e5be124d7c603e73b996e02291c88d758ccb33918cef71a8996e648b86ad"} Oct 31 00:18:27.873478 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.873143 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"6a3d2c4f2dc6f4faa496312ebf0a3701c3fec1fe06695ed2d352215caad04f60"} Oct 31 00:18:27.873478 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.873179 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"7b41049bf702f8997ef2e5d9a50abaa09a7fdd7fb86777bf43c424fdeb098d99"} Oct 31 00:18:27.873478 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.873220 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"2f6bf20beeab9899db3c9efaeddef3de897798b02b2265f35822953e0f95f96a"} Oct 31 00:18:27.873478 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.873260 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"dee44bdf63c21edfc34abf55ae6e5dbde0df56eb56877462a7cfdf4edd041a3d"} Oct 31 00:18:27.875788 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.875758 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" event={"ID":"171f86cc-abaf-4fec-8ca2-d4809d731a37","Type":"ContainerStarted","Data":"13dbb301c7eff8707b9777639582a496962f4d00792537bf154372bd51178c97"} Oct 31 00:18:27.923999 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:27.922769 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rnzgf" podStartSLOduration=8.442663201 podStartE2EDuration="17.922752286s" totalImagesPullingTime="9.480089085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:27.921803986 +0000 UTC m=+17.696829830" watchObservedRunningTime="2026-10-31 00:18:27.922752286 +0000 UTC m=+17.697778131" Oct 31 00:18:28.554503 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.554477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.554517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.554569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554609 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554662 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret podName:5ef4d6fd-df80-4242-9971-19f9ff9316c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:44.554647785 +0000 UTC m=+34.329673639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret") pod "global-pull-secret-syncer-77tnr" (UID: "5ef4d6fd-df80-4242-9971-19f9ff9316c5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554681 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554687 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554697 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:28.554712 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554708 2576 projected.go:196] Error preparing data for projected volume kube-api-access-g8rwd for pod openshift-network-diagnostics/network-check-target-q2pbw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:28.554993 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:44.554722643 +0000 UTC m=+34.329748471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:28.554993 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.554753 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd podName:45a542cb-bea5-4d21-9208-07c3712bda6e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:44.55474691 +0000 UTC m=+34.329772731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8rwd" (UniqueName: "kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd") pod "network-check-target-q2pbw" (UID: "45a542cb-bea5-4d21-9208-07c3712bda6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:28.673922 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.673886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:28.673922 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.673908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:28.673922 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:28.673922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:28.674152 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.674013 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:28.674152 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.674099 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:28.674279 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:28.674154 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:29.883449 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:29.883421 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"03cc293d03eb7b3b3cc0c4f10b6903863df38f817c1b78c316cbea931b5daa10"} Oct 31 00:18:30.674917 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:30.674890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:30.675086 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:30.674974 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:30.675086 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:30.675051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:30.675177 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:30.675144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:30.675235 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:30.675184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:30.675299 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:30.675282 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:32.674382 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.674202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:32.674947 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.674233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:32.674947 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:32.674445 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:32.674947 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.674269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:32.674947 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:32.674537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:32.674947 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:32.674588 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:32.893189 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.893162 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" event={"ID":"90bc26e1-5d51-46e9-8832-d12588bc7ac3","Type":"ContainerStarted","Data":"f4783b657650ae992edb1a30c9f69676c8e66eaf83ef1fb8c59503058e2292df"} Oct 31 00:18:32.893433 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.893412 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:32.908100 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.908081 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:32.987395 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:32.987315 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" podStartSLOduration=7.561453296 podStartE2EDuration="21.987304245s" totalImagesPullingTime="14.425850949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:32.986903739 +0000 UTC m=+22.761929582" watchObservedRunningTime="2026-10-31 00:18:32.987304245 +0000 UTC m=+22.762330087" Oct 31 00:18:33.234796 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.234766 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:33.896329 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.896300 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:33.911757 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.911731 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:33.976693 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.976658 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7mtxv"] Oct 31 00:18:33.976851 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.976814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:33.976956 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:33.976931 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:33.979529 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.979505 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2pbw"] Oct 31 00:18:33.979617 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.979604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:33.979684 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:33.979670 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:33.980005 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.979989 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77tnr"] Oct 31 00:18:33.980076 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:33.980065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:33.980153 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:33.980139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:35.673798 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:35.673764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:35.674289 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:35.673764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:35.674289 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:35.673866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:35.674289 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:35.673784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:35.674289 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:35.673919 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:35.674289 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:35.674033 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:37.673649 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:37.673619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:37.674096 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:37.673628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:37.674096 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:37.673706 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:37.674096 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:37.673721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:37.674096 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:37.673796 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:37.674096 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:37.673874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:39.673945 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:39.673917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:39.673945 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:39.673934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:39.674411 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:39.674004 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2pbw" podUID="45a542cb-bea5-4d21-9208-07c3712bda6e" Oct 31 00:18:39.674411 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:39.674059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:39.674411 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:39.674128 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mtxv" podUID="e59fd416-1856-4d1d-a234-93fa7c4575db" Oct 31 00:18:39.674411 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:39.674201 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77tnr" podUID="5ef4d6fd-df80-4242-9971-19f9ff9316c5" Oct 31 00:18:40.011569 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.011542 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeReady" Oct 31 00:18:40.011673 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.011639 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:18:40.062583 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.062559 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-msmzl"] Oct 31 00:18:40.091307 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.091287 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kw88c"] Oct 31 00:18:40.091419 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.091406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:40.111443 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.111418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:40.111703 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.111454 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:18:40.119894 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.115696 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.119894 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.116353 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.131763 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.131738 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:18:40.131877 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.131860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.147740 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.147588 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxts4"] Oct 31 00:18:40.147740 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.147721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.149963 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.149942 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:18:40.150077 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.150026 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-94d5g\"" Oct 31 00:18:40.150188 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.150171 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:18:40.150288 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.150233 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:18:40.171495 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.171477 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm"] Oct 31 00:18:40.171611 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.171597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.184859 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.184842 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qb96d"] Oct 31 00:18:40.184992 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.184943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.188541 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.188526 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q7dwz"] Oct 31 00:18:40.188652 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.188638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.191482 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.191464 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr"] Oct 31 00:18:40.191612 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.191571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:40.194880 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.194865 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d476j"] Oct 31 00:18:40.195000 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.194985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.197797 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.197781 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4zwc2"] Oct 31 00:18:40.197885 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.197871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.200843 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.200776 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7"] Oct 31 00:18:40.200911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.200897 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.214176 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214158 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.214268 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214178 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.214375 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214364 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.214375 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214369 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:18:40.214483 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214408 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.214483 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214425 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:18:40.214483 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214450 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:18:40.214483 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214469 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.214661 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.214567 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.219323 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.219306 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55f45b7759-45twl"] Oct 31 00:18:40.219448 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.219431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.228713 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.228694 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:18:40.230158 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230132 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:18:40.230360 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230342 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.230544 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230521 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.230622 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230555 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:18:40.230681 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230624 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:18:40.230789 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.230777 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:18:40.236295 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.236280 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw"] Oct 31 00:18:40.236406 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.236392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.239754 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.239735 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5"] Oct 31 00:18:40.239892 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.239824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:40.243747 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.243726 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:18:40.247387 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.247370 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-scmpl"] Oct 31 00:18:40.247477 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.247464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.247841 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.247820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.247939 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.247856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.247939 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.247882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.248036 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248036 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248119 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.248119 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qmm\" (UniqueName: \"kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm\") pod \"kas-connection-checker-685895c7c9-kw88c\" (UID: \"193040bf-22d1-4852-acfe-b2b3dd70b1a8\") " pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:40.248181 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4blr\" (UniqueName: \"kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr\") pod \"kas-connection-checker-685895c7c9-msmzl\" (UID: \"9b22cf76-b18b-4ce7-84e3-0c4940594b5c\") " pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:40.248246 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248246 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248308 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248308 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.248308 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248391 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248391 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248391 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248391 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248391 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248521 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.248521 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.248411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.254064 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.253991 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:18:40.254064 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.253994 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:18:40.254201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254066 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:18:40.254201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254053 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:18:40.254201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254185 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:18:40.254201 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254215 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254229 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254277 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ftjzd\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254289 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254302 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254305 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254337 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c22vv\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254341 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m9w9q\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254338 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5gqw7\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254365 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g79rw\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254373 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:18:40.254385 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254381 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hlbxq\"" Oct 31 00:18:40.254762 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.254413 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q7p8f\"" Oct 31 00:18:40.257155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.257140 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lwb8l\"" Oct 31 00:18:40.264149 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.264133 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:18:40.264433 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.264419 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nkt5k\"" Oct 31 00:18:40.272155 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.272140 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-msmzl"] Oct 31 00:18:40.272251 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.272173 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kw88c"] Oct 31 00:18:40.272251 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.272196 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl"] Oct 31 00:18:40.272402 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.272265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.289469 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289454 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:18:40.289540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289479 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxts4"] Oct 31 00:18:40.289540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289496 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm"] Oct 31 00:18:40.289540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289509 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qb96d"] Oct 31 00:18:40.289540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289521 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:18:40.289540 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289532 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr"] Oct 31 00:18:40.289681 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289549 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw"] Oct 31 00:18:40.289681 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:40.289681 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289561 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d476j"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289700 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289721 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289735 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f45b7759-45twl"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289750 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q7dwz"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289762 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl"] Oct 31 00:18:40.289775 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289776 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scmpl"] Oct 31 00:18:40.289950 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.289793 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4zwc2"] Oct 31 00:18:40.292354 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.292340 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:18:40.314920 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.314900 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315011 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.314942 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.315011 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.314972 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315109 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315064 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315109 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315099 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.315185 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315172 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:18:40.315239 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315181 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:18:40.315298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315262 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315269 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315298 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315289 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.315411 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315313 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:18:40.315411 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315330 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.315512 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315436 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.315512 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315492 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:18:40.315512 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.315495 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:18:40.348695 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.348800 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.348800 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvbv\" (UniqueName: \"kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.348800 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.348887 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.348845 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:40.348887 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.348943 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.348910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.848894747 +0000 UTC m=+30.623920573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:40.349013 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.349013 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.348974 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:40.349013 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.348990 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:40.349013 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.348987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xvq\" (UniqueName: \"kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.349215 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hd66\" (UniqueName: \"kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.349215 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.349043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.849030367 +0000 UTC m=+30.624056188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:40.349215 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.349215 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qmm\" (UniqueName: \"kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm\") pod \"kas-connection-checker-685895c7c9-kw88c\" (UID: \"193040bf-22d1-4852-acfe-b2b3dd70b1a8\") " pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:40.349215 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.349437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.349437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.349437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.349437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.349437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.349571 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.349571 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gb9\" (UniqueName: \"kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9\") pod \"kas-connection-checker-685895c7c9-q7dwz\" (UID: \"d7dc021f-37cc-43d4-a4e2-50e449ff0032\") " pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:40.349625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.349625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.349625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.349625 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kff94\" (UniqueName: \"kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.349739 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.349739 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.349739 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.349848 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswhd\" (UniqueName: \"kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.349848 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.349947 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dt8g\" (UniqueName: \"kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.349947 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.349947 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.350084 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.350084 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.349988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.350084 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw\") pod \"volume-data-source-validator-58c9f5f46d-hrljw\" (UID: \"c491ada7-6f46-4e40-97f0-699cc84dc60d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:40.350084 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.350291 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.350291 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.350291 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqk7\" (UniqueName: \"kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.350441 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.350441 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.350441 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.350441 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4blr\" (UniqueName: \"kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr\") pod \"kas-connection-checker-685895c7c9-msmzl\" (UID: \"9b22cf76-b18b-4ce7-84e3-0c4940594b5c\") " pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:40.350441 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.350692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.350692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.350692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.350692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.350692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.350911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhm4r\" (UniqueName: \"kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.350911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.350911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.350911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.350911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.350984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351044 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\"" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.351125 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ntw\" (UniqueName: \"kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.351405 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.351405 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351157 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8bqtd\"" Oct 31 00:18:40.351405 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.351405 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.351359 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:40.351405 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.351374 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:40.351595 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.351433 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.851418277 +0000 UTC m=+30.626444113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:40.351640 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.351804 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.351744 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.354223 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.354193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.354294 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.354235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.354294 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.354273 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.354538 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.354521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.379796 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.379766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4blr\" (UniqueName: \"kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr\") pod \"kas-connection-checker-685895c7c9-msmzl\" (UID: \"9b22cf76-b18b-4ce7-84e3-0c4940594b5c\") " pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:40.380383 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.380351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.380466 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.380435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.381816 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.381793 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.383166 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.383130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.383888 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.383868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qmm\" (UniqueName: \"kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm\") pod \"kas-connection-checker-685895c7c9-kw88c\" (UID: \"193040bf-22d1-4852-acfe-b2b3dd70b1a8\") " pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:40.419937 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.419920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:40.436571 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.436552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:40.452461 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvbv\" (UniqueName: \"kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.452461 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.452461 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.452610 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xvq\" (UniqueName: \"kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.452659 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.452612 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:40.452659 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.452648 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.952636832 +0000 UTC m=+30.727662666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:40.452755 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hd66\" (UniqueName: \"kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.452755 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.452755 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.452897 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.452897 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.452897 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gb9\" (UniqueName: \"kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9\") pod \"kas-connection-checker-685895c7c9-q7dwz\" (UID: \"d7dc021f-37cc-43d4-a4e2-50e449ff0032\") " pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:40.452897 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.452897 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kff94\" (UniqueName: \"kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.452982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswhd\" (UniqueName: \"kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dt8g\" (UniqueName: \"kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.453146 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw\") pod \"volume-data-source-validator-58c9f5f46d-hrljw\" (UID: \"c491ada7-6f46-4e40-97f0-699cc84dc60d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqk7\" (UniqueName: \"kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453329 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhm4r\" (UniqueName: \"kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fkp\" (UniqueName: \"kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp\") pod \"network-check-source-6f78d59f99-fjxrl\" (UID: \"886edf34-51d0-43c5-84c3-a1996c07aacb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.455556 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ntw\" (UniqueName: \"kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.453889 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.453944 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.953926013 +0000 UTC m=+30.728951835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.453989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.454102 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.454156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.954141511 +0000 UTC m=+30.729167347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.454251 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.454289 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.954276234 +0000 UTC m=+30.729302069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.454626 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.95461203 +0000 UTC m=+30.729637879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:40.456057 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.456546 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.456546 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.454933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.456546 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.455278 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.456546 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.455472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.456546 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.455704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.456931 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.456914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.457063 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.457042 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.457244 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.457223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.457681 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.457665 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.458346 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.458322 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.477813 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.477782 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvbv\" (UniqueName: \"kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.485868 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.485821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hd66\" (UniqueName: \"kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.485868 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.485858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswhd\" (UniqueName: \"kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.486246 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.486193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqk7\" (UniqueName: \"kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.486332 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.486244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gb9\" (UniqueName: \"kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9\") pod \"kas-connection-checker-685895c7c9-q7dwz\" (UID: \"d7dc021f-37cc-43d4-a4e2-50e449ff0032\") " pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:40.486376 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.486338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhm4r\" (UniqueName: \"kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.486471 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.486454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kff94\" (UniqueName: \"kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.486792 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.486772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xvq\" (UniqueName: \"kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.490558 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.490535 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw\") pod \"volume-data-source-validator-58c9f5f46d-hrljw\" (UID: \"c491ada7-6f46-4e40-97f0-699cc84dc60d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:40.491306 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.491285 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ntw\" (UniqueName: \"kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.492269 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.492249 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dt8g\" (UniqueName: \"kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.506398 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.506284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:40.530973 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.530815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:40.539062 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.539002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:40.546703 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.546678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:40.553970 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.553949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fkp\" (UniqueName: \"kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp\") pod \"network-check-source-6f78d59f99-fjxrl\" (UID: \"886edf34-51d0-43c5-84c3-a1996c07aacb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:40.587807 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.585124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fkp\" (UniqueName: \"kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp\") pod \"network-check-source-6f78d59f99-fjxrl\" (UID: \"886edf34-51d0-43c5-84c3-a1996c07aacb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:40.612383 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.612344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:40.614443 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.614377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:40.620077 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.619868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:40.620077 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.619946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:40.815320 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.815294 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kw88c"] Oct 31 00:18:40.817256 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.817227 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-msmzl"] Oct 31 00:18:40.856703 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.856674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:40.856813 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.856748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:40.856813 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.856782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:40.856962 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.856946 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:40.857018 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.856966 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:40.857018 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857010 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.856998917 +0000 UTC m=+31.632024751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:40.857118 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857075 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:40.857118 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857085 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:40.857118 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857117 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.857107176 +0000 UTC m=+31.632133017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:40.857285 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857172 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:40.857285 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.857221 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.857195012 +0000 UTC m=+31.632220847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:40.883805 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.883777 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qb96d"] Oct 31 00:18:40.884910 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.884887 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34220d9_9132_4909_a542_35d25dd1af45.slice/crio-7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196 WatchSource:0}: Error finding container 7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196: Status 404 returned error can't find the container with id 7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196 Oct 31 00:18:40.898517 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.898484 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q7dwz"] Oct 31 00:18:40.898908 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.898890 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7dc021f_37cc_43d4_a4e2_50e449ff0032.slice/crio-cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a WatchSource:0}: Error finding container cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a: Status 404 returned error can't find the container with id cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a Oct 31 00:18:40.904371 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.904348 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7"] Oct 31 00:18:40.904465 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.904415 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494ca01f_586a_4d2b_8625_5e56046ecf86.slice/crio-431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4 WatchSource:0}: Error finding container 431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4: Status 404 returned error can't find the container with id 431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4 Oct 31 00:18:40.911802 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.911519 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4zwc2"] Oct 31 00:18:40.913434 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.913411 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9796d2d_3c0c_4565_a1c3_e8b5b584f97a.slice/crio-42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f WatchSource:0}: Error finding container 42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f: Status 404 returned error can't find the container with id 42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f Oct 31 00:18:40.913969 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.913891 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" event={"ID":"494ca01f-586a-4d2b-8625-5e56046ecf86","Type":"ContainerStarted","Data":"431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4"} Oct 31 00:18:40.918771 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.918714 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-msmzl" event={"ID":"9b22cf76-b18b-4ce7-84e3-0c4940594b5c","Type":"ContainerStarted","Data":"819c7c0a5aed78ba909cf391db7718760824123b123d1b1f1c155d1b27f2f193"} Oct 31 00:18:40.921247 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.921145 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" event={"ID":"d34220d9-9132-4909-a542-35d25dd1af45","Type":"ContainerStarted","Data":"7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196"} Oct 31 00:18:40.924597 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.924457 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kw88c" event={"ID":"193040bf-22d1-4852-acfe-b2b3dd70b1a8","Type":"ContainerStarted","Data":"07919e1da87f9ebef1288f28d77acb4a2de3b0e64fc09f4c7e8616ec39dc67b5"} Oct 31 00:18:40.925753 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.925698 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" event={"ID":"d7dc021f-37cc-43d4-a4e2-50e449ff0032","Type":"ContainerStarted","Data":"cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a"} Oct 31 00:18:40.958023 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.957994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:40.958121 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.958081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:40.958177 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.958161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.958287 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.958254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:40.958350 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.958302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:40.958447 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958432 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.958414152 +0000 UTC m=+31.733439994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:40.958541 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958528 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:40.958602 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958572 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.958559133 +0000 UTC m=+31.733584955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:40.958662 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958628 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:40.958662 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958659 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.958647411 +0000 UTC m=+31.733673240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:40.958760 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958710 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:40.958760 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958740 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.958730563 +0000 UTC m=+31.733756398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:40.958858 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958792 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:40.958858 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:40.958822 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:41.958813014 +0000 UTC m=+31.733838838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:40.964414 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.963963 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw"] Oct 31 00:18:40.966132 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.966098 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc491ada7_6f46_4e40_97f0_699cc84dc60d.slice/crio-2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1 WatchSource:0}: Error finding container 2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1: Status 404 returned error can't find the container with id 2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1 Oct 31 00:18:40.999188 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:40.999152 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode356e65d_d1de_4fb4_8a77_0707aacf1999.slice/crio-7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b WatchSource:0}: Error finding container 7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b: Status 404 returned error can't find the container with id 7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b Oct 31 00:18:40.999805 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:40.999553 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scmpl"] Oct 31 00:18:41.000761 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:41.000735 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5e6bd8_67a3_4a2c_a636_2ba8a4315a0d.slice/crio-bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434 WatchSource:0}: Error finding container bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434: Status 404 returned error can't find the container with id bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434 Oct 31 00:18:41.001036 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.000984 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5"] Oct 31 00:18:41.014595 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.014576 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl"] Oct 31 00:18:41.018871 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:41.018849 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886edf34_51d0_43c5_84c3_a1996c07aacb.slice/crio-c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab WatchSource:0}: Error finding container c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab: Status 404 returned error can't find the container with id c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab Oct 31 00:18:41.673457 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.673420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:41.673913 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.673898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:41.674357 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.674340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:41.692996 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.692975 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:18:41.693070 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.693040 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:18:41.693111 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.693070 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\"" Oct 31 00:18:41.693111 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.693109 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\"" Oct 31 00:18:41.693287 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.693184 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\"" Oct 31 00:18:41.867561 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.867525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:41.867993 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.867718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:41.867993 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.867781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:41.867993 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.867956 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:41.867993 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.867974 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:41.868228 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868032 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.868012016 +0000 UTC m=+33.643037863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:41.868307 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868286 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:41.868307 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868302 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:41.868408 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868358 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.868341897 +0000 UTC m=+33.643367732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:41.868487 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868474 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:41.868535 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.868527 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.868513645 +0000 UTC m=+33.643539482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:41.935499 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.934717 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-msmzl" event={"ID":"9b22cf76-b18b-4ce7-84e3-0c4940594b5c","Type":"ContainerStarted","Data":"22b13326d6e7a9c1d128e841dfbfd6cef5aed70ac8514a915d6528036b79ef59"} Oct 31 00:18:41.938082 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.937650 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kw88c" event={"ID":"193040bf-22d1-4852-acfe-b2b3dd70b1a8","Type":"ContainerStarted","Data":"28619d1ea16b2d8d826644be7a78589c1489372cf77c8ed43ee18992de672b9e"} Oct 31 00:18:41.943762 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.943673 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" event={"ID":"886edf34-51d0-43c5-84c3-a1996c07aacb","Type":"ContainerStarted","Data":"c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab"} Oct 31 00:18:41.946460 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.946019 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" event={"ID":"d7dc021f-37cc-43d4-a4e2-50e449ff0032","Type":"ContainerStarted","Data":"cbde6ab14826f33de17500034598ff422a1b544b6867b23c97992424b466cc0e"} Oct 31 00:18:41.948867 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.948794 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scmpl" event={"ID":"e356e65d-d1de-4fb4-8a77-0707aacf1999","Type":"ContainerStarted","Data":"7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b"} Oct 31 00:18:41.951831 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.951770 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" event={"ID":"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d","Type":"ContainerStarted","Data":"bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434"} Oct 31 00:18:41.953964 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.953894 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" event={"ID":"c491ada7-6f46-4e40-97f0-699cc84dc60d","Type":"ContainerStarted","Data":"2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1"} Oct 31 00:18:41.956159 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.956136 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" event={"ID":"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a","Type":"ContainerStarted","Data":"42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f"} Oct 31 00:18:41.969236 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.969041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:41.969236 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.969124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:41.969236 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.969171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:41.969236 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.969197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:41.969442 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.969274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:41.969442 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969399 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:41.969532 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969450 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.969434161 +0000 UTC m=+33.744459985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969747 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.969729668 +0000 UTC m=+33.744755504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969753 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.969782696 +0000 UTC m=+33.744808532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969807 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.969839492 +0000 UTC m=+33.744865316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969868 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:41.969923 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:41.969902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:43.969891267 +0000 UTC m=+33.744917115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:41.980931 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:41.980885 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-msmzl" podStartSLOduration=283.980870239 podStartE2EDuration="4m43.980870239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:41.97940646 +0000 UTC m=+31.754432326" watchObservedRunningTime="2026-10-31 00:18:41.980870239 +0000 UTC m=+31.755896081" Oct 31 00:18:42.099651 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:42.098527 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kw88c" podStartSLOduration=284.098503085 podStartE2EDuration="4m44.098503085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:42.031070015 +0000 UTC m=+31.806095891" watchObservedRunningTime="2026-10-31 00:18:42.098503085 +0000 UTC m=+31.873528930" Oct 31 00:18:42.099651 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:42.099305 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" podStartSLOduration=284.099297489 podStartE2EDuration="4m44.099297489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:42.097665436 +0000 UTC m=+31.872691280" watchObservedRunningTime="2026-10-31 00:18:42.099297489 +0000 UTC m=+31.874323332" Oct 31 00:18:43.890500 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.890459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.890557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.890595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890630 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890654 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890717 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.890698814 +0000 UTC m=+37.665724652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890750 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890766 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890815 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.890799435 +0000 UTC m=+37.665825262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890883 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:43.890999 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.890915 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.890904612 +0000 UTC m=+37.665930438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:43.968112 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.968080 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" event={"ID":"c491ada7-6f46-4e40-97f0-699cc84dc60d","Type":"ContainerStarted","Data":"d3a8a37b016a2a88aefe5ca423d83a7dff28df647e1c05ac1a462caeaf1ee528"} Oct 31 00:18:43.991692 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.991650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:43.991852 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.991733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:43.991852 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.991794 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:43.991852 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.991820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:43.991852 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.991850 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.991831517 +0000 UTC m=+37.766857353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:43.992079 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.991886 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:43.992079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.991900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:43.992079 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.991921 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.991910639 +0000 UTC m=+37.766936479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:43.992079 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:43.991950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:43.992079 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.991967 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.991956855 +0000 UTC m=+37.766982690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:43.992415 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.992160 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:43.992415 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.992230 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.992202833 +0000 UTC m=+37.767228662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:43.992415 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.992299 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:43.992415 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:43.992335 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.99232341 +0000 UTC m=+37.767349232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:44.014760 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.014711 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" podStartSLOduration=251.72059129 podStartE2EDuration="4m14.014692532s" totalImagesPullingTime="2.294101242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:44.012695595 +0000 UTC m=+33.787721440" watchObservedRunningTime="2026-10-31 00:18:44.014692532 +0000 UTC m=+33.789718376" Oct 31 00:18:44.599036 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.599003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:44.599233 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.599065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:44.599613 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.599585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:44.599759 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:44.599739 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:18:44.599828 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:44.599814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:19:16.599792596 +0000 UTC m=+66.374818447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : secret "metrics-daemon-secret" not found Oct 31 00:18:44.603609 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.603580 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:44.603780 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.603760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:44.739562 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.739501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:44.759660 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:44.759631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:45.203516 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.203455 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77tnr"] Oct 31 00:18:45.204558 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:45.204520 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef4d6fd_df80_4242_9971_19f9ff9316c5.slice/crio-0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56 WatchSource:0}: Error finding container 0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56: Status 404 returned error can't find the container with id 0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56 Oct 31 00:18:45.216325 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:45.216291 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a542cb_bea5_4d21_9208_07c3712bda6e.slice/crio-fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a WatchSource:0}: Error finding container fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a: Status 404 returned error can't find the container with id fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a Oct 31 00:18:45.217995 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.217932 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2pbw"] Oct 31 00:18:45.991748 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.991654 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/0.log" Oct 31 00:18:45.991748 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.991733 2576 generic.go:356] "Generic (PLEG): container finished" podID="d34220d9-9132-4909-a542-35d25dd1af45" containerID="7eb5bf6132c9974cfcf174212eeac0f3420cb2ffa9db049ca140a0fba6f664a3" exitCode=255 Oct 31 00:18:45.991951 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.991843 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" event={"ID":"d34220d9-9132-4909-a542-35d25dd1af45","Type":"ContainerDied","Data":"7eb5bf6132c9974cfcf174212eeac0f3420cb2ffa9db049ca140a0fba6f664a3"} Oct 31 00:18:45.993789 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.993202 2576 scope.go:122] "RemoveContainer" containerID="7eb5bf6132c9974cfcf174212eeac0f3420cb2ffa9db049ca140a0fba6f664a3" Oct 31 00:18:45.994836 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.994786 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77tnr" event={"ID":"5ef4d6fd-df80-4242-9971-19f9ff9316c5","Type":"ContainerStarted","Data":"0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56"} Oct 31 00:18:45.996822 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.996799 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2pbw" event={"ID":"45a542cb-bea5-4d21-9208-07c3712bda6e","Type":"ContainerStarted","Data":"fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a"} Oct 31 00:18:45.999138 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:45.998776 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scmpl" event={"ID":"e356e65d-d1de-4fb4-8a77-0707aacf1999","Type":"ContainerStarted","Data":"f096baae480bf9a73b44f4a93b03bb0ee1fb6754c66f0707c130649499b647d0"} Oct 31 00:18:46.001248 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.001006 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" event={"ID":"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d","Type":"ContainerStarted","Data":"b915e683767d31779d772a3454a764ca07295f74a9374f726affd415bd33905c"} Oct 31 00:18:46.003231 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.003195 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" event={"ID":"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a","Type":"ContainerStarted","Data":"e6b863146d3b9837251cf19ec32be4eb12a19537a098c84b520077597e15e374"} Oct 31 00:18:46.139388 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.139330 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" podStartSLOduration=248.437821543 podStartE2EDuration="4m13.139313039s" totalImagesPullingTime="4.701491496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:46.09217864 +0000 UTC m=+35.867204486" watchObservedRunningTime="2026-10-31 00:18:46.139313039 +0000 UTC m=+35.914338885" Oct 31 00:18:46.186853 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.186784 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-scmpl" podStartSLOduration=1.469667966 podStartE2EDuration="6.186765528s" totalImagesPullingTime="4.717097562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:46.186181217 +0000 UTC m=+35.961207066" watchObservedRunningTime="2026-10-31 00:18:46.186765528 +0000 UTC m=+35.961791370" Oct 31 00:18:46.188253 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.188197 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" podStartSLOduration=251.916477577 podStartE2EDuration="4m16.188184885s" totalImagesPullingTime="4.271707308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:46.14099387 +0000 UTC m=+35.916019713" watchObservedRunningTime="2026-10-31 00:18:46.188184885 +0000 UTC m=+35.963210729" Oct 31 00:18:46.303437 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.303414 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qxclf"] Oct 31 00:18:46.342866 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.342776 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qxclf"] Oct 31 00:18:46.342931 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.342895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.366367 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.364874 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:18:46.366367 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.365093 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:18:46.366367 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.365197 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\"" Oct 31 00:18:46.377818 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.376717 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:18:46.395818 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.395460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.398222 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.398181 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:18:46.520433 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.520433 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g54\" (UniqueName: \"kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.520627 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.520627 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.520627 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.520627 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.520815 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.520815 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.520815 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.520701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.621398 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.621541 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.621541 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.621652 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.621652 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g54\" (UniqueName: \"kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.621751 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.621751 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.621846 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.621846 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.621800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.622070 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.622041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.622183 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.622162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.622312 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:46.622296 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:46.622366 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:46.622360 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.122340749 +0000 UTC m=+36.897366580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:46.622442 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.622425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.622554 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:46.622536 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:18:46.622627 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:46.622597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:47.122579218 +0000 UTC m=+36.897605055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : configmap "kube-rbac-proxy" not found Oct 31 00:18:46.622736 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.622712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.623746 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.623721 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.648985 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.648960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g54\" (UniqueName: \"kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:46.649672 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.649542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.744126 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.744092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:46.832285 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.831946 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd"] Oct 31 00:18:46.864782 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.861819 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg"] Oct 31 00:18:46.865919 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.865402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:46.879587 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.877374 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd"] Oct 31 00:18:46.879587 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.877417 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg"] Oct 31 00:18:46.879587 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.877506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:46.887234 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.886370 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qlrh5\"" Oct 31 00:18:46.887234 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.886433 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:46.887234 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:46.886474 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:18:47.009534 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.009497 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" event={"ID":"494ca01f-586a-4d2b-8625-5e56046ecf86","Type":"ContainerStarted","Data":"348098d016f8afe2eded6dc6ecdd4e372c37219e3a94c192baddea37ec1fbe70"} Oct 31 00:18:47.023369 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.023341 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:47.024407 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.024386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/0.log" Oct 31 00:18:47.024589 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.024568 2576 generic.go:356] "Generic (PLEG): container finished" podID="d34220d9-9132-4909-a542-35d25dd1af45" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" exitCode=255 Oct 31 00:18:47.024783 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.024655 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" event={"ID":"d34220d9-9132-4909-a542-35d25dd1af45","Type":"ContainerDied","Data":"b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c"} Oct 31 00:18:47.024890 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.024871 2576 scope.go:122] "RemoveContainer" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" Oct 31 00:18:47.025017 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.025005 2576 scope.go:122] "RemoveContainer" containerID="7eb5bf6132c9974cfcf174212eeac0f3420cb2ffa9db049ca140a0fba6f664a3" Oct 31 00:18:47.025325 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.025071 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:47.025992 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.025970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8tbl\" (UniqueName: \"kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl\") pod \"migrator-66df5d496b-gmllg\" (UID: \"abf37dbf-0b0a-4b33-a3a6-07be5096b090\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:47.026644 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.026593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bzr\" (UniqueName: \"kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr\") pod \"migrator-66df5d496b-sqlxd\" (UID: \"a194a3a1-a93d-44ad-a58c-8d5b8edef450\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:47.061037 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.060997 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" podStartSLOduration=248.763330528 podStartE2EDuration="4m14.060981203s" totalImagesPullingTime="5.297650675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:47.060108598 +0000 UTC m=+36.835134444" watchObservedRunningTime="2026-10-31 00:18:47.060981203 +0000 UTC m=+36.836007047" Oct 31 00:18:47.127971 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.127942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bzr\" (UniqueName: \"kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr\") pod \"migrator-66df5d496b-sqlxd\" (UID: \"a194a3a1-a93d-44ad-a58c-8d5b8edef450\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:47.128142 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.128123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:47.128388 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.128336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:47.128467 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.128434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tbl\" (UniqueName: \"kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl\") pod \"migrator-66df5d496b-gmllg\" (UID: \"abf37dbf-0b0a-4b33-a3a6-07be5096b090\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:47.128994 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.128975 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:47.129091 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.129013 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:18:47.129091 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.129023 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:48.129010549 +0000 UTC m=+37.904036370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:47.129091 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.129061 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:48.129046049 +0000 UTC m=+37.904071876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : configmap "kube-rbac-proxy" not found Oct 31 00:18:47.149120 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.149098 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:18:47.156583 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.156562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tbl\" (UniqueName: \"kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl\") pod \"migrator-66df5d496b-gmllg\" (UID: \"abf37dbf-0b0a-4b33-a3a6-07be5096b090\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:47.157408 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.157363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bzr\" (UniqueName: \"kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr\") pod \"migrator-66df5d496b-sqlxd\" (UID: \"a194a3a1-a93d-44ad-a58c-8d5b8edef450\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:47.195876 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.195711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:47.220096 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.220077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:47.554276 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.554247 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd"] Oct 31 00:18:47.554928 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:47.554902 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda194a3a1_a93d_44ad_a58c_8d5b8edef450.slice/crio-8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b WatchSource:0}: Error finding container 8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b: Status 404 returned error can't find the container with id 8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b Oct 31 00:18:47.591911 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.591891 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg"] Oct 31 00:18:47.594859 ip-10-0-132-66 kubenswrapper[2576]: W1031 00:18:47.594822 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf37dbf_0b0a_4b33_a3a6_07be5096b090.slice/crio-54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3 WatchSource:0}: Error finding container 54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3: Status 404 returned error can't find the container with id 54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3 Oct 31 00:18:47.936200 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.936159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:47.936384 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.936236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:47.936384 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936350 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936386 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936402 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:47.936405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936423 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:55.936404034 +0000 UTC m=+45.711429859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936451 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:55.936435688 +0000 UTC m=+45.711461510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936473 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:47.936484 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936483 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:47.936793 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:47.936517 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:55.936506307 +0000 UTC m=+45.711532161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:48.029232 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.029188 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ts89r-l8pft" event={"ID":"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a","Type":"ContainerStarted","Data":"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81"} Oct 31 00:18:48.029398 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.029252 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ts89r-l8pft" event={"ID":"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a","Type":"ContainerStarted","Data":"935d5beca79532b36673b5e71bd49610743b05ae9b67142d8f96333f78172a8b"} Oct 31 00:18:48.030452 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.030433 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" event={"ID":"abf37dbf-0b0a-4b33-a3a6-07be5096b090","Type":"ContainerStarted","Data":"54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3"} Oct 31 00:18:48.031551 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.031528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" event={"ID":"a194a3a1-a93d-44ad-a58c-8d5b8edef450","Type":"ContainerStarted","Data":"8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b"} Oct 31 00:18:48.032932 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.032912 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:48.033217 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.033189 2576 scope.go:122] "RemoveContainer" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" Oct 31 00:18:48.033389 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.033372 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:48.034352 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.034331 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" event={"ID":"886edf34-51d0-43c5-84c3-a1996c07aacb","Type":"ContainerStarted","Data":"4920987a30eab21004e7f3d46cc3032f6e260046abbfd998ac4842be5b92d53a"} Oct 31 00:18:48.036780 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.036762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:48.036863 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.036817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:48.036901 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.036890 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:48.036929 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.036895 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:48.036929 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.036926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.036914311 +0000 UTC m=+45.811940146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:48.036993 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.036960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:48.036993 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.036981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:48.037050 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.037000086 +0000 UTC m=+45.812025907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:48.037050 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.037041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:48.037050 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037047 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:48.037140 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.037067565 +0000 UTC m=+45.812093405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:48.037140 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037093 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.037087559 +0000 UTC m=+45.812113380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:48.037140 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037122 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:48.037264 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.037154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.037146903 +0000 UTC m=+45.812172729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:48.081785 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.081715 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ts89r-l8pft" podStartSLOduration=2.0816946 podStartE2EDuration="2.0816946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:48.080068243 +0000 UTC m=+37.855094090" watchObservedRunningTime="2026-10-31 00:18:48.0816946 +0000 UTC m=+37.856720443" Oct 31 00:18:48.128561 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.128507 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" podStartSLOduration=241.903211752 podStartE2EDuration="4m8.12849007s" totalImagesPullingTime="6.225278318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:48.127860461 +0000 UTC m=+37.902886321" watchObservedRunningTime="2026-10-31 00:18:48.12849007 +0000 UTC m=+37.903515912" Oct 31 00:18:48.138672 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.138326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:48.138672 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:48.138540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:48.138672 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.138602 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:48.138672 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.138641 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:18:48.138672 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.138652 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.138637287 +0000 UTC m=+39.913663133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:48.138948 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:48.138693 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.138678939 +0000 UTC m=+39.913704760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : configmap "kube-rbac-proxy" not found Oct 31 00:18:49.042345 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:49.042297 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2pbw" event={"ID":"45a542cb-bea5-4d21-9208-07c3712bda6e","Type":"ContainerStarted","Data":"20bde272b0eb53fd9a721bca68b0d44e3fe2ae4ad1bcbbb8da0db5a0eee2186b"} Oct 31 00:18:49.093375 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:49.092285 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q2pbw" podStartSLOduration=35.992331809 podStartE2EDuration="39.092269307s" totalImagesPullingTime="3.099937498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:49.091557283 +0000 UTC m=+38.866583154" watchObservedRunningTime="2026-10-31 00:18:49.092269307 +0000 UTC m=+38.867295150" Oct 31 00:18:49.773143 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:49.773055 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9cp4_e127939f-dcd5-405f-97c6-87b7a68a5c36/dns-node-resolver/0.log" Oct 31 00:18:50.047430 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.047345 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" event={"ID":"abf37dbf-0b0a-4b33-a3a6-07be5096b090","Type":"ContainerStarted","Data":"4f01f04f70f1b06999dd55d9a194bca54193105db9daf1f40ffccb8c405f82a9"} Oct 31 00:18:50.049239 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.048967 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" event={"ID":"a194a3a1-a93d-44ad-a58c-8d5b8edef450","Type":"ContainerStarted","Data":"c59c69fa2ea85c2fdab0f5828323c3d058b4e88af11a8437673d0379a875b792"} Oct 31 00:18:50.049239 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.049095 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:50.104896 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.104839 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" podStartSLOduration=2.301909429 podStartE2EDuration="4.104825138s" totalImagesPullingTime="1.802915709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:50.104453784 +0000 UTC m=+39.879479627" watchObservedRunningTime="2026-10-31 00:18:50.104825138 +0000 UTC m=+39.879850982" Oct 31 00:18:50.155191 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.155137 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" podStartSLOduration=2.235012547 podStartE2EDuration="4.155122844s" totalImagesPullingTime="1.920110297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:50.15343131 +0000 UTC m=+39.928457154" watchObservedRunningTime="2026-10-31 00:18:50.155122844 +0000 UTC m=+39.930148687" Oct 31 00:18:50.160162 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.160116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:50.160308 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:50.160241 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:50.160308 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:50.160294 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:18:50.160378 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:50.160319 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.160299452 +0000 UTC m=+43.935325290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:50.160378 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:50.160341 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.160331083 +0000 UTC m=+43.935356905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : configmap "kube-rbac-proxy" not found Oct 31 00:18:50.160378 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.160241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:50.507275 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.507235 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:50.507439 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.507303 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:50.507682 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.507666 2576 scope.go:122] "RemoveContainer" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" Oct 31 00:18:50.507937 ip-10-0-132-66 kubenswrapper[2576]: E1031 00:18:50.507909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:50.974907 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:50.974884 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-krnph_c5adc70d-63ec-4ab8-9ca5-8fb4928040fc/node-ca/0.log" Oct 31 00:18:51.376165 ip-10-0-132-66 kubenswrapper[2576]: I1031 00:18:51.376121 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scmpl_e356e65d-d1de-4fb4-8a77-0707aacf1999/serve-healthcheck-canary/0.log" Oct 31 00:18:51.468295 ip-10-0-132-66 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:18:51.486622 ip-10-0-132-66 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:18:51.486788 ip-10-0-132-66 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:18:51.486838 ip-10-0-132-66 systemd[1]: kubelet.service: Consumed 2.946s CPU time. Oct 31 00:18:51.506801 ip-10-0-132-66 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:18:51.589049 ip-10-0-132-66 kubenswrapper[5011]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:51.589049 ip-10-0-132-66 kubenswrapper[5011]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:18:51.589049 ip-10-0-132-66 kubenswrapper[5011]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:51.589049 ip-10-0-132-66 kubenswrapper[5011]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:51.591060 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.590991 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591062 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591068 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591071 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591075 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591078 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591081 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591084 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591086 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591089 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591092 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591094 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591097 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591100 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591103 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.591098 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591106 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591109 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591111 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591114 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591117 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591120 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591122 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591125 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591127 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591130 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591133 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591144 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591148 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591151 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591154 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591157 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591160 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591162 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591165 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.591502 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591167 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591171 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591173 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591176 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591179 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591181 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591184 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591187 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591190 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591193 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591196 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591199 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591202 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591219 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591224 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591228 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591232 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591236 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591240 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591244 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.591960 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591248 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591251 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591253 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591256 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591260 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591263 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591266 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591269 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591271 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591274 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591277 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591279 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591282 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591284 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591287 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591290 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591292 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591295 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591298 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591300 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.592551 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591303 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591306 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591309 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591311 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591315 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591317 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591320 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591323 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591326 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591329 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591331 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591334 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591337 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591339 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591342 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591344 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591347 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591355 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591358 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591361 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.593125 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591364 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591366 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591368 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591371 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591374 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591376 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591378 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591381 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591384 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591387 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591389 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.591392 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592160 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592167 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592170 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592173 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592176 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592179 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592181 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.593831 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592186 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592191 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592195 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592198 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592201 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592220 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592224 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592227 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592229 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592232 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592235 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592238 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592241 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592244 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592246 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592249 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592251 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592254 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592257 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592260 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.594495 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592263 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592266 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592268 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592271 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592274 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592276 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592279 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592281 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592284 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592287 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592290 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592292 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592296 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592299 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592301 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592304 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592307 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592309 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592312 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592314 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.595178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592317 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592319 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592322 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592324 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592327 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592330 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592333 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592335 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592337 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592340 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592343 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592345 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592348 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592350 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592353 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592356 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592358 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592361 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592364 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.595734 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592367 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592369 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592372 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592375 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592377 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592380 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592382 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592385 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592388 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592391 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592394 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592396 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592399 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592401 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592404 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592423 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592427 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592431 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592434 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592437 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.596233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592439 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592442 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592445 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592447 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592450 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592453 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592456 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592459 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592461 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592464 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592468 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592472 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592474 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592477 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592480 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592483 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592486 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592488 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592492 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.596708 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592494 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592959 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592964 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592967 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592970 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592973 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592976 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592980 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592983 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592987 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592991 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592995 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.592999 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593002 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593006 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593016 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593019 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593023 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593027 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.597224 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593031 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593035 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593039 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593043 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593047 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593051 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593055 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593058 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593062 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593067 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593071 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593075 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593079 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593083 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593086 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593091 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593095 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593099 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593103 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593106 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.597829 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593110 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593114 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593119 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593123 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593127 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593133 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593138 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593142 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593148 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593152 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593156 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593160 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593164 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593169 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593173 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593177 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593183 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593188 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593193 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593197 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.598648 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593201 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593219 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593223 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593228 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593232 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593237 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593241 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593246 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593250 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593255 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593259 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593263 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593268 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593271 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593275 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593279 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593283 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593287 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593292 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.599515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593296 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593304 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593310 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593316 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593320 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593326 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593330 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593334 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593338 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593342 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593347 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593352 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593356 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593361 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593365 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593370 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593375 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593380 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593387 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.600331 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593391 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593395 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593400 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593404 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593408 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593412 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593415 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593418 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593420 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.593423 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593540 5011 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593556 5011 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593562 5011 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593568 5011 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593575 5011 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593578 5011 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593583 5011 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593588 5011 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593591 5011 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593594 5011 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593597 5011 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593601 5011 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:18:51.601168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593604 5011 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593607 5011 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593610 5011 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593613 5011 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593617 5011 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593620 5011 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593624 5011 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593627 5011 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593630 5011 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593633 5011 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593637 5011 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593641 5011 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593644 5011 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593648 5011 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593651 5011 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593654 5011 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593657 5011 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593660 5011 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593663 5011 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593666 5011 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593671 5011 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593674 5011 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593677 5011 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593680 5011 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593685 5011 flags.go:64] FLAG: --enable-server="true" Oct 31 00:18:51.602160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593688 5011 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593692 5011 flags.go:64] FLAG: --event-burst="100" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593700 5011 flags.go:64] FLAG: --event-qps="50" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593703 5011 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593706 5011 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593709 5011 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593714 5011 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593717 5011 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593720 5011 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593723 5011 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593727 5011 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593730 5011 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593733 5011 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593735 5011 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593738 5011 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593741 5011 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593744 5011 flags.go:64] FLAG: --feature-gates="" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593748 5011 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593752 5011 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593755 5011 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593758 5011 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593761 5011 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593764 5011 flags.go:64] FLAG: --help="false" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593767 5011 flags.go:64] FLAG: --hostname-override="ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593770 5011 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:18:51.603340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593773 5011 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593776 5011 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593780 5011 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593783 5011 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593786 5011 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593789 5011 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593793 5011 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593796 5011 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593800 5011 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593805 5011 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593808 5011 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593811 5011 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593814 5011 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593817 5011 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593820 5011 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593822 5011 flags.go:64] FLAG: --lock-file="" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593825 5011 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593828 5011 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593832 5011 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593838 5011 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593841 5011 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593844 5011 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593847 5011 flags.go:64] FLAG: --logging-format="text" Oct 31 00:18:51.604430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593850 5011 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593853 5011 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593856 5011 flags.go:64] FLAG: --manifest-url="" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593859 5011 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593864 5011 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593867 5011 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593870 5011 flags.go:64] FLAG: --max-pods="110" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593873 5011 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593876 5011 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593879 5011 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593883 5011 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593886 5011 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593889 5011 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593892 5011 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593899 5011 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593902 5011 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593906 5011 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593909 5011 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593914 5011 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593917 5011 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593921 5011 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593924 5011 flags.go:64] FLAG: --port="10250" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593927 5011 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593929 5011 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06caefb75075e1bab" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593933 5011 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593935 5011 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:18:51.605422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593939 5011 flags.go:64] FLAG: --register-node="true" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593941 5011 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593945 5011 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593948 5011 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593951 5011 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593954 5011 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593958 5011 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593961 5011 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593964 5011 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593966 5011 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593969 5011 flags.go:64] FLAG: --runonce="false" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593972 5011 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593975 5011 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593978 5011 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593981 5011 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593984 5011 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593987 5011 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593990 5011 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593993 5011 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593996 5011 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.593999 5011 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594002 5011 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594005 5011 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594008 5011 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594013 5011 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:18:51.606530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594016 5011 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594022 5011 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594025 5011 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594028 5011 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594032 5011 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594035 5011 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594038 5011 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594041 5011 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594044 5011 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594047 5011 flags.go:64] FLAG: --v="2" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594051 5011 flags.go:64] FLAG: --version="false" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594056 5011 flags.go:64] FLAG: --vmodule="" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594060 5011 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594063 5011 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594253 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594262 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594267 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594271 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594275 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594279 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594283 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594287 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594291 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.607616 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594295 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594300 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594304 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594308 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594312 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594315 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594320 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594324 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594332 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594337 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594340 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594343 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594345 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594348 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594351 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594355 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594360 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594363 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594366 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.608315 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594369 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594371 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594375 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594377 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594380 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594382 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594387 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594391 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594394 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594396 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594399 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594401 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594404 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594407 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594409 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594411 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594414 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594417 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594419 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.608923 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594423 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594426 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594430 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594433 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594436 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594439 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594441 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594444 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594447 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594450 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594452 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594455 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594457 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594460 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594462 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594465 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594467 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594470 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594473 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594475 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.609434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594478 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594480 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594483 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594486 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594488 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594491 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594493 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594496 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594499 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594502 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594505 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594507 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594518 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594522 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594528 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594532 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594536 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594540 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594544 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594548 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.609920 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594553 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594557 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594561 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594565 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594569 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594573 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594577 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594581 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594585 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594589 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594594 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594598 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594602 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594606 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594610 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594614 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594618 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594623 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.610520 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594627 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.594672 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594727 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594734 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594739 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594744 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594749 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594756 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594762 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594769 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594775 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594781 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594787 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594792 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594797 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594801 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.611007 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594805 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594809 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594813 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594817 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594820 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594824 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594828 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594832 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594837 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594841 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594844 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594848 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594853 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594857 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594861 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594866 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594868 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594871 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594874 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594876 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.611497 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594879 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594881 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594884 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594889 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594892 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594895 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594898 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594900 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594904 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594907 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594910 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594913 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594916 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594919 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594921 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594924 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594927 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594930 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594933 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594935 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.611981 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594938 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594940 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594943 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594946 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594948 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594950 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594953 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594956 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594959 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594961 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594964 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594966 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594969 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594972 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594974 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594978 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594982 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594984 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594987 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.612468 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594990 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594993 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.594995 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595002 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595005 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595008 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595011 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595013 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595016 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595019 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595022 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595024 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595027 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595030 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595032 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595035 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595037 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595040 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595042 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595045 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.612916 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595047 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595050 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595053 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595055 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595057 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595060 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595062 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595065 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595069 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595073 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595077 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595082 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.595085 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.600594 5011 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.600626 5011 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.600658 5011 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.600666 5011 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600754 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.613405 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600764 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600771 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600778 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600783 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600789 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600793 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600798 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600802 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600807 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600811 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600815 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600820 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600824 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600830 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600837 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600844 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600850 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600854 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600858 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.614072 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600863 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600867 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600871 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600876 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600880 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600884 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600888 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600893 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600897 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600902 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600907 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600912 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600916 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600921 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600925 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600930 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600934 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600938 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600943 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600947 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.614857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600952 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600957 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600961 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600966 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600970 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600975 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600980 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600985 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600989 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600993 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.600997 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601002 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601006 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601011 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601015 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601019 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601024 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601028 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601033 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601037 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.615426 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601041 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601046 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601051 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601055 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601060 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601065 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601069 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601073 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601078 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601082 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601086 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601091 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601095 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601100 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601104 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601108 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601113 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601117 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601121 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.615907 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601126 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601130 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601135 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601139 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601144 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601148 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601153 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601157 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601161 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601166 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601170 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601174 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601178 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601182 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601186 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601190 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601194 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601199 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601219 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.616410 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601224 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601237 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601242 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601246 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601250 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601254 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601259 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601264 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.601304 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601552 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601562 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601568 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601573 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601577 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601583 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601587 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.616860 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601591 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601596 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601600 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601605 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601609 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601615 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601619 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601623 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601628 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601632 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601636 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601640 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601644 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601648 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601653 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601658 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601662 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601666 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601671 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.617264 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601675 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601679 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601683 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601687 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601693 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601697 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601702 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601706 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601713 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601718 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601724 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601729 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601735 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601740 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601744 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601749 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601753 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601758 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601762 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.617713 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601766 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601771 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601775 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601780 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601784 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601788 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601793 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601797 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601801 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601806 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601810 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601814 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601818 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601823 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601827 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601831 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601836 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601841 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601845 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601849 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.618263 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601853 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601857 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601862 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601866 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601871 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601875 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601880 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601885 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601889 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601893 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601898 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601902 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601906 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601911 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601916 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601920 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601924 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601928 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601932 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601937 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.618754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601941 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601945 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601949 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601953 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601957 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601962 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601966 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601970 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601977 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601982 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601987 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601991 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601995 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.601999 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602004 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602008 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602012 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602016 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602021 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.619262 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602026 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602030 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.602066 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602120 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602127 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602132 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602137 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602141 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602145 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602150 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602154 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602158 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602162 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602166 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602170 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602174 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:51.619744 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602178 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602182 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602186 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602190 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602195 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602199 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602219 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602224 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602228 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602233 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602237 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602241 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602245 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602249 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602254 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602258 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602262 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602267 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602272 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602276 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:51.620123 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602280 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602285 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602289 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602293 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602297 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602301 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602306 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602310 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602314 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602318 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602323 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602327 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602331 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602335 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602339 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602344 5011 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602348 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602352 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602356 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602361 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:51.620628 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602365 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602371 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602380 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602386 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602390 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602394 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602399 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602403 5011 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602407 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602411 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602416 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602421 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602425 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602429 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602434 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602439 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602443 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602447 5011 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602452 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:51.621138 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602457 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602462 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602466 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602470 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602474 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602479 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602484 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602488 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602492 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602496 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602501 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602505 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602509 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602523 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602527 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602534 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602539 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602543 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602547 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:51.621596 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602551 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602556 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602560 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602564 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602569 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602574 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602578 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602583 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602587 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602591 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602599 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602606 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602611 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602615 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:51.602619 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.602700 5011 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604189 5011 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604273 5011 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:18:51.622077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604540 5011 server.go:1014] "Starting client certificate rotation" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604673 5011 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604725 5011 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:10 +0000 UTC" deadline="2028-08-06 06:28:27.686440102 +0000 UTC" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.604759 5011 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15486h9m36.081683158s" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.605315 5011 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.607021 5011 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.608399 5011 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.610308 5011 log.go:25] "Validated CRI v1 image API" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.611033 5011 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:18:51.622525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.613001 5011 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e306abcc-6c4a-478a-a64a-9cbc6ffe5afe:/dev/nvme0n1p3 fc862f49-e223-411d-9db6-7023170324ea:/dev/nvme0n1p4] Oct 31 00:18:51.622784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.613015 5011 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07919e1da87f9ebef1288f28d77acb4a2de3b0e64fc09f4c7e8616ec39dc67b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07919e1da87f9ebef1288f28d77acb4a2de3b0e64fc09f4c7e8616ec39dc67b5/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/289dd2627806f2e7f1ecb589309cf1a577c70c0d7a716941d92bca680f8400da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/289dd2627806f2e7f1ecb589309cf1a577c70c0d7a716941d92bca680f8400da/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/819c7c0a5aed78ba909cf391db7718760824123b123d1b1f1c155d1b27f2f193/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/819c7c0a5aed78ba909cf391db7718760824123b123d1b1f1c155d1b27f2f193/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/935d5beca79532b36673b5e71bd49610743b05ae9b67142d8f96333f78172a8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/935d5beca79532b36673b5e71bd49610743b05ae9b67142d8f96333f78172a8b/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a/volumes/kubernetes.io~projected/kube-api-access-w4n8d:{mountpoint:/var/lib/kubelet/pods/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a/volumes/kubernetes.io~projected/kube-api-access-w4n8d major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b/volumes/kubernetes.io~projected/kube-api-access-8hd66:{mountpoint:/var/lib/kubelet/pods/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b/volumes/kubernetes.io~projected/kube-api-access-8hd66 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/171f86cc-abaf-4fec-8ca2-d4809d731a37/volumes/kubernetes.io~projected/kube-api-access-5kkgh:{mountpoint:/var/lib/kubelet/pods/171f86cc-abaf-4fec-8ca2-d4809d731a37/volumes/kubernetes.io~projected/kube-api-access-5kkgh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/193040bf-22d1-4852-acfe-b2b3dd70b1a8/volumes/kubernetes.io~projected/kube-api-access-r6qmm:{mountpoint:/var/lib/kubelet/pods/193040bf-22d1-4852-acfe-b2b3dd70b1a8/volumes/kubernetes.io~projected/kube-api-access-r6qmm major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~projected/kube-api-access-gqnrk:{mountpoint:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~projected/kube-api-access-gqnrk major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bbdb593-002d-4500-8f49-9a3208c8388f/volumes/kubernetes.io~projected/kube-api-access-96g54:{mountpoint:/var/lib/kubelet/pods/2bbdb593-002d-4500-8f49-9a3208c8388f/volumes/kubernetes.io~projected/kube-api-access-96g54 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a/volumes/kubernetes.io~projected/kube-api-access-lhqk7:{mountpoint:/var/lib/kubelet/pods/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a/volumes/kubernetes.io~projected/kube-api-access-lhqk7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45a542cb-bea5-4d21-9208-07c3712bda6e/volumes/kubernetes.io~projected/kube-api-access-g8rwd:{mountpoint:/var/lib/kubelet/pods/45a542cb-bea5-4d21-9208-07c3712bda6e/volumes/kubernetes.io~projected/kube-api-access-g8rwd major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45b345b2-4e5d-4510-8d14-17dacce01a56/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/45b345b2-4e5d-4510-8d14-17dacce01a56/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1/volumes/kubernetes.io~projected/kube-api-access-v82nk:{mountpoint:/var/lib/kubelet/pods/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1/volumes/kubernetes.io~projected/kube-api-access-v82nk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~projected/kube-api-access-44ntw:{mountpoint:/var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~projected/kube-api-access-44ntw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ef4d6fd-df80-4242-9971-19f9ff9316c5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5ef4d6fd-df80-4242-9971-19f9ff9316c5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/886edf34-51d0-43c5-84c3-a1996c07aacb/volumes/kubernetes.io~projected/kube-api-access-64fkp:{mountpoint:/var/lib/kubelet/pods/886edf34-51d0-43c5-84c3-a1996c07aacb/volumes/kubernetes.io~projected/kube-api-access-64fkp major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~projected/kube-api-access-nr8qj:{mountpoint:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~projected/kube-api-access-nr8qj major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b22cf76-b18b-4ce7-84e3-0c4940594b5c/volumes/kubernetes.io~projected/kube-api-access-t4blr:{mountpoint:/var/lib/kubelet/pods/9b22cf76-b18b-4ce7-84e3-0c4940594b5c/volumes/kubernetes.io~projected/kube-api-access-t4blr major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a194a3a1-a93d-44ad-a58c-8d5b8edef450/volumes/kubernetes.io~projected/kube-api-access-n6bzr:{mountpoint:/var/lib/kubelet/pods/a194a3a1-a93d-44ad-a58c-8d5b8edef450/volumes/kubernetes.io~projected/kube-api-access-n6bzr major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~projected/kube-api-access-2dt8g:{mountpoint:/var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~projected/kube-api-access-2dt8g major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf37dbf-0b0a-4b33-a3a6-07be5096b090/volumes/kubernetes.io~projected/kube-api-access-z8tbl:{mountpoint:/var/lib/kubelet/pods/abf37dbf-0b0a-4b33-a3a6-07be5096b090/volumes/kubernetes.io~projected/kube-api-access-z8tbl major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c491ada7-6f46-4e40-97f0-699cc84dc60d/volumes/kubernetes.io~projected/kube-api-access-s6srw:{mountpoint:/var/lib/kubelet/pods/c491ada7-6f46-4e40-97f0-699cc84dc60d/volumes/kubernetes.io~projected/kube-api-access-s6srw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc/volumes/kubernetes.io~projected/kube-api-access-5kgpn:{mountpoint:/var/lib/kubelet/pods/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc/volumes/kubernetes.io~projected/kube-api-access-5kgpn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~projected/kube-api-access-zhm4r:{mountpoint:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~projected/kube-api-access-zhm4r major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/default-certificate major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~projected/kube-api-access-kff94:{mountpoint:/var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~projected/kube-api-access-kff94 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5574e12-afad-4304-b163-f30046d15fc1/volumes/kubernetes.io~projected/kube-api-access-nttdf:{mountpoint:/var/lib/kubelet/pods/d5574e12-afad-4304-b163-f30046d15fc1/volumes/kubernetes.io~projected/kube-api-access-nttdf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7dc021f-37cc-43d4-a4e2-50e449ff0032/volumes/kubernetes.io~projected/kube-api-access-88gb9:{mountpoint:/var/lib/kubelet/pods/d7dc021f-37cc-43d4-a4e2-50e449ff0032/volumes/kubernetes.io~projected/kube-api-access-88gb9 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25/volumes/kubernetes.io~projected/kube-api-access-x99nn:{mountpoint:/var/lib/kubelet/pods/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25/volumes/kubernetes.io~projected/kube-api-access-x99nn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e127939f-dcd5-405f-97c6-87b7a68a5c36/volumes/kubernetes.io~projected/kube-api-access-hcq4h:{mountpoint:/var/lib/kubelet/pods/e127939f-dcd5-405f-97c6-87b7a68a5c36/volumes/kubernetes.io~projected/kube-api-access-hcq4h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~projected/kube-api-access-xdvbv:{mountpoint:/var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~projected/kube-api-access-xdvbv major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e59fd416-1856-4d1d-a234-93fa7c4575db/volumes/kubernetes.io~projected/kube-api-access-qvpwk:{mountpoint:/var/lib/kubelet/pods/e59fd416-1856-4d1d-a234-93fa7c4575db/volumes/kubernetes.io~projected/kube-api-access-qvpwk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/kube-api-access-jfqvb:{mountpoint:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/kube-api-access-jfqvb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee655651-2201-4c07-9b83-10440b2ed2c2/volumes/kubernetes.io~projected/kube-api-access-b6xvq:{mountpoint:/var/lib/kubelet/pods/ee655651-2201-4c07-9b83-10440b2ed2c2/volumes/kubernetes.io~projected/kube-api-access-b6xvq major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~projected/kube-api-access-qswhd:{mountpoint:/var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~projected/kube-api-access-qswhd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/kube-api-access-66qms:{mountpoint:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/kube-api-access-66qms major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9c9d53fa3d318f3d4fdc97c77dcdd56d11d530d4de06cd191f18e893ac0ff5b2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/28952dfefca7089687ae5a77e8118cd4560a6787c2a2bb0d9414512f91be5c8f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5d3eed2205e60d27ed06a9dbcb05c9a57bba7dec88abb5197f988e1c1c9ca52c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ad04f6d33213f810fd61aefab55be9a4a5b1ee518cf35b2b438cc60067484114/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e75ba3239cc0b9a973c7d44be1ccce503e4c42c5f7ef26154dfd0afc1e2b93d3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/04372695c829e2918b3d117d28e40684d310caafe8a291852f0fd15c850a183a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f46ca28bba2ca8f5bdedc54da3c1ceaecb2eff476398c1664e4b836a3451541c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/af8bb7f18a864cacbbc9457f1ca8eb09a57071c72389cc872dda45ed60ca13e2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/06365d4501f245ae093e3647b6b7a6bb7f2e6138fbaa9f70ada7389a600b089e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d1d215a939829896b3abac82e700c37f960d8f898268ac646eb965c8679d86ca/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/75d5decc3b7c0cb552832acb3ed8a4bb8c9c00a08ab48b88f69196cd5c87fa2e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a1f3592545b2e07f5fcab6077a6357254b2c0f9921fd247ed7c7cf74c78eeec4/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e7fe8b4fccddc004193eabd88917640910bea6553d363784d13fde5c51d3638b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b395267f0bfdec9966f9f4736f9c72312a2ab4b9f63baa52162d95f897885e9e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2313905e6877f482b7b221e843e6d3f2b690ad0f721cc114318e077ee4c722e6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1614eb0181d37bd0974834b0968a113863529651b7d7be6283f85b682c119ade/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/900f13995039284f4a2a249d09fd4aeca301f3925d2a999a1cd7512749209c1e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7fd6770fc434d75374e5238ecf380a4354d89fd917b7cb827cdba100ab7add08/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/363fe7f6ff773e6695e5e1f5fc27b45ff8a7051c0763e9fb00707c5fb123891e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a2ec9d24e5592bc0f7163979f850e6121e09812029330680dae21e81b02415a0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/613d0e6190f4a7c99e16aac03ea17d2e66556618ba9f31e60d9eedfe6aa24097/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a3a8ba9b4a15f8d34fb24849eb57997f98afb66efa19d9bfc70adea237955114/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6edbee62aa7b9e20b3b35955ab54928d845d0244d82bf53946b43a53587c8eeb/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f6919af2f4ddf1007e645ae30e8751a4a08e75d5e845d3ffabf669b1b348e8e7/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a57a7e2fbeb6d380669f30e8be0ff09bd4b60b89058123b85e5f9de842bc8d69/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/76c5378ac25fd0b5bdd9dccea7e04ca189282514ba8191e7dfa0d43fd867508b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/fcb78efbd266d855650018fd123cda656d518219da349ae54e32212872c11967/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f2261309dea46ffd49fb09b1e552a46764b87086299c29db4dd9c7cdd7c8de6b/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/7cefeea163a509d5b13c3999c48797eb4512d4060f33cc65441ae45606755b90/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ff60959b808937d97f93eb80830236b2729a62a38afe77f6ac9a466af762887a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/a8ad44b631303451523c4dcff2fb3edfaf1c3289fac1bd76c56eaab6ea43a69d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/4ffb1a0cc61978dfe16f874e4c43996a1faaef2efdaf375ffa81356871bd457e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/66eeb06b7a6087abfe7b6c597b2876f548f6a427faa9a81a848e261d1bbc4c92/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/d9fa88c3cc0044222f8863fddf356f4940f2f3bd6f44af413e9f4b3ad39b4318/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/68c0f03243f5695a8304a30006be1a937158887e130ce88f57e5f45eb3840165/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/aab825573682e49182b5e8b91a247ec9090f3002cb08e667124a609653d18c64/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/6641974f104e3d190648226186ab5e8251bdc8dcac81e5d912f2376a03f1e337/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c467e040de063f976737c7890959125652799424063633e6cf9fb20821fea2c9/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ecbe7bc4b85c05838af62b2cf7a96cb3c05c251d9ab6cbbf01a2d1073a9076b1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/79e252159e43905440bab0eda43f749f19544688c01cd891f6b82659f13a9ec7/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/77b57b4f015b9e9981329da372ea3105af8ed3b3ff21eaf9725adca80f09ea2a/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/95e4cb629ecd83de158d6d75894fc69b5c73d8fbd11ca5b127cf457dbc936480/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/fb8abf499303af2a2b875e39bf8de101dd6060b85fd6e26bc8e2346602ff88a0/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/7a92f81df3a52a0aba89dc13b78ee9e5d96f2abf26b62a1f73b79f94240d3db1/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/0de4794426d8657e2705eebd8e99c66a8fdc3982f8b8ec161d408e19c2a672e0/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/01bfeef888f29efbe8256ca4ad33575a81e25d2fffb41d9de55d4e9143c27ab8/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/fcd3c250745031720d4beba57d02fc7d81a1d949d5310e082332d9cf58d476d7/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/1b5c954dca986c62d36221c2034043c377fddbde23aea9b724d59b40182cbaf4/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/2971f0baadbd9ba0d8a451ac3b72bfa46db5a105b256d45da0dfba5ad53b6454/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/8547af01b29ffd068ee4a15aeba7c26123b3c735a8ea49a23d4d6f650e187a59/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/94b2119e76266b74fa9e788db48cd7e27f0d9bdc7fb2314471e4e89a2c1dc343/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c001347bb1c9170f722305ef74eddfa763330fbfd15db345395c25ef2e3dc95f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6baed1860dda672c883001288aea8d52f5f84517d30136bd8be347bbb6035290/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2bcea2849f1ab7ef8e9eabf76662618c15d4fceebbabb24461a0f27aed30f2be/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/4c95f8abf80f317dbc0de795e91253fabaa89fec6f19d51a95545bb008a26905/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/5f201b4b47c43c1776224ccbc4e59936de030984c4871d31bba9556f79f24d50/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4bf45ce4decd375bcf572f7548be251fa38b2dbac1b605b7a2ca7acb3e248857/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/386cc59cbf1c6aa5fa32e6078290cd573f9ca86f8099bd63d72e876964e4d0c0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/74b65ce2d9d8320c6aa53c45f081f74a910ed95a55e1e245efbc5eb7616e7c43/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8120629662b04fe6cb7da170a992cdce1b2fd7a1c135d691771f95efa214c0a1/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:18:51.623006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.622149 5011 manager.go:217] Machine: {Timestamp:2026-10-31 00:18:51.621688778 +0000 UTC m=+0.070776409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099526 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f02a0e181c03a3deaa164a9db614 SystemUUID:ec22f02a-0e18-1c03-a3de-aa164a9db614 BootID:2f193b13-9f07-4fdf-b63c-98e3da3391fe Filesystems:[{Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~projected/kube-api-access-qswhd DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b/volumes/kubernetes.io~projected/kube-api-access-8hd66 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c491ada7-6f46-4e40-97f0-699cc84dc60d/volumes/kubernetes.io~projected/kube-api-access-s6srw DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/666cf6bb69e9e949978885a1fbfa9a465320f8cc8d5e5ba1ad4f3183f365c314/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1/volumes/kubernetes.io~projected/kube-api-access-v82nk DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b22cf76-b18b-4ce7-84e3-0c4940594b5c/volumes/kubernetes.io~projected/kube-api-access-t4blr DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~projected/kube-api-access-zhm4r DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/193040bf-22d1-4852-acfe-b2b3dd70b1a8/volumes/kubernetes.io~projected/kube-api-access-r6qmm DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~projected/kube-api-access-kff94 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42850331b78e576c9e3fb663267bad67d02ea3904221aca6a099fd824c648c5f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d34220d9-9132-4909-a542-35d25dd1af45/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/819c7c0a5aed78ba909cf391db7718760824123b123d1b1f1c155d1b27f2f193/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~projected/kube-api-access-2dt8g DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c85281a30e8bc9563607aac1232864b9a9498143e013b42cd35771f0f9c69cab/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7494fbd8d8a25816a987b131cf3bc0147966a7690a46806651be2da86d4d228b/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5574e12-afad-4304-b163-f30046d15fc1/volumes/kubernetes.io~projected/kube-api-access-nttdf DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7dc021f-37cc-43d4-a4e2-50e449ff0032/volumes/kubernetes.io~projected/kube-api-access-88gb9 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee655651-2201-4c07-9b83-10440b2ed2c2/volumes/kubernetes.io~projected/kube-api-access-b6xvq DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes/kubernetes.io~projected/kube-api-access-66qms DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a/volumes/kubernetes.io~projected/kube-api-access-lhqk7 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e127939f-dcd5-405f-97c6-87b7a68a5c36/volumes/kubernetes.io~projected/kube-api-access-hcq4h DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~projected/kube-api-access-44ntw DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7109ffedd00d7ed7fa26b16fba93af65ed04d925b8db8b85ac5f8d33752e6196/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abf37dbf-0b0a-4b33-a3a6-07be5096b090/volumes/kubernetes.io~projected/kube-api-access-z8tbl DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54d1ce4f8daaeac5441217bd6b053996773031111b65004b48d1a1da86069dc3/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/171f86cc-abaf-4fec-8ca2-d4809d731a37/volumes/kubernetes.io~projected/kube-api-access-5kkgh DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/90bc26e1-5d51-46e9-8832-d12588bc7ac3/volumes/kubernetes.io~projected/kube-api-access-nr8qj DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a194a3a1-a93d-44ad-a58c-8d5b8edef450/volumes/kubernetes.io~projected/kube-api-access-n6bzr DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07919e1da87f9ebef1288f28d77acb4a2de3b0e64fc09f4c7e8616ec39dc67b5/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/886edf34-51d0-43c5-84c3-a1996c07aacb/volumes/kubernetes.io~projected/kube-api-access-64fkp DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf3af31195f577c5ac1e33e3012aedb2cf229ef88d71d2bbe8f8318c92aa3434/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087b4cd1c1ab69ad97d22cc38f2fc15dbb41215e6262d5270d59133057f5a4ab/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bbdb593-002d-4500-8f49-9a3208c8388f/volumes/kubernetes.io~projected/kube-api-access-96g54 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d99eed91bdfba614ba2ad65c13c54af01b916542cadd9964ed0bbe0c138923b/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbfaf409051ea201fdf16d724f6d5244a5b6123d975657dbe0a007942b5302ca/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bd6ca45c59f4896c1898fbb34278f91493da431fcb2a220011ca62a29281288/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/431b75c44635e3566214793c7af6520ba230fbd728e715fcace27209378e11e4/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d4543164e09b67ffe939fbf858cd7440cbe33a8ffb6a0b28a0fe3ca59d20da1/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29dcb073-2ad3-4d21-91e2-24e2536de375/volumes/kubernetes.io~projected/kube-api-access-gqnrk DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25/volumes/kubernetes.io~projected/kube-api-access-x99nn DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a/volumes/kubernetes.io~projected/kube-api-access-w4n8d DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d65622f4de80d0d8db879bc2c938fdf792f257afc03ebcd79cc9d2cb8458da9/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~projected/kube-api-access-jfqvb DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~projected/kube-api-access-xdvbv DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf7bce523d006e536e74d9e19fcf9ad3c6cfbadccd555a7644125affb4311d6a/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45a542cb-bea5-4d21-9208-07c3712bda6e/volumes/kubernetes.io~projected/kube-api-access-g8rwd DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e59fd416-1856-4d1d-a234-93fa7c4575db/volumes/kubernetes.io~projected/kube-api-access-qvpwk DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcbaa32aed1b9b2cae652f5bff623e5551e972031ab47557ddf84c73f7e99d56/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ef4d6fd-df80-4242-9971-19f9ff9316c5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45b345b2-4e5d-4510-8d14-17dacce01a56/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a726109b2743b7b50ac1d45296f0320f8b3edc13230e874b51ac62da807d6a/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe46f8f4e69d570581c09e0aba15c356577cc6062027358dd4f56ad45e27961a/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0470f45ada9c1190e87409052d80b702c2da59b553b30249488b7abc124a1d56/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc/volumes/kubernetes.io~projected/kube-api-access-5kgpn DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/289dd2627806f2e7f1ecb589309cf1a577c70c0d7a716941d92bca680f8400da/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e356e65d-d1de-4fb4-8a77-0707aacf1999/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/494ca01f-586a-4d2b-8625-5e56046ecf86/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/935d5beca79532b36673b5e71bd49610743b05ae9b67142d8f96333f78172a8b/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0470f45ada9c119 MacAddress:5e:6e:65:e5:fb:b2 Speed:10000 Mtu:8901} {Name:07919e1da87f9eb MacAddress:92:e2:ca:36:4b:b5 Speed:10000 Mtu:8901} {Name:2d4543164e09b67 MacAddress:36:66:a0:44:9d:65 Speed:10000 Mtu:8901} {Name:42850331b78e576 MacAddress:ce:c5:07:5e:7f:8a Speed:10000 Mtu:8901} {Name:431b75c44635e35 MacAddress:da:32:28:05:cc:71 Speed:10000 Mtu:8901} {Name:54d1ce4f8daaeac MacAddress:ca:68:b6:4a:c1:19 Speed:10000 Mtu:8901} {Name:7109ffedd00d7ed MacAddress:5e:8f:ae:cf:a9:3c Speed:10000 Mtu:8901} {Name:7494fbd8d8a2581 MacAddress:4e:eb:fd:7e:6a:a6 Speed:10000 Mtu:8901} {Name:819c7c0a5aed78b MacAddress:3a:0d:fb:35:f8:9c Speed:10000 Mtu:8901} {Name:8d99eed91bdfba6 MacAddress:ca:f5:23:75:a4:17 Speed:10000 Mtu:8901} {Name:935d5beca79532b MacAddress:26:2c:b9:0e:bb:89 Speed:10000 Mtu:8901} {Name:bf3af31195f577c MacAddress:82:f2:aa:0e:d3:b2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:f3:29:e7 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:94:80:75:6d:7e Speed:0 Mtu:8901} {Name:c85281a30e8bc95 MacAddress:e2:46:d6:9c:6a:d4 Speed:10000 Mtu:8901} {Name:cf7bce523d006e5 MacAddress:ce:ae:e1:3e:8a:e6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:f3:29:e7 Speed:0 Mtu:9001} {Name:fe46f8f4e69d570 MacAddress:06:38:3a:41:d5:9c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:72:46:9e:4d:9b:4c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:f4:b1:cb:72:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:18:51.623006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.622543 5011 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:18:51.623006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.622587 5011 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:18:51.623006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.622903 5011 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.622924 5011 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623041 5011 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623048 5011 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623058 5011 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623077 5011 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623095 5011 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:18:51.623280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623251 5011 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623356 5011 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623396 5011 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623405 5011 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623416 5011 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623425 5011 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:18:51.623472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.623433 5011 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:18:51.624133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.624120 5011 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:18:51.624133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.624132 5011 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:51.624649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.624630 5011 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:18:51.627708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.627694 5011 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:18:51.627708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.627708 5011 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:51.627808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.627799 5011 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:18:51.627977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.627966 5011 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:18:51.628006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.627995 5011 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:18:51.628096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628087 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:18:51.628135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628101 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:18:51.628135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628108 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:18:51.628135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628114 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:18:51.628135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628124 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:18:51.628135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628134 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628142 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628148 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628155 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628161 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628179 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628189 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628226 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:18:51.628338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.628233 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:18:51.631880 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.631866 5011 server.go:1263] "Started kubelet" Oct 31 00:18:51.632519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.632276 5011 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:18:51.632519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.632381 5011 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:18:51.632656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.632581 5011 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:18:51.632623 ip-10-0-132-66 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:18:51.632794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.632704 5011 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:18:51.634859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.634839 5011 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:18:51.637540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637525 5011 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637550 5011 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.637628 5011 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-66.ec2.internal\" not found" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637705 5011 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:12 +0000 UTC" deadline="2028-08-16 14:32:29.203681434 +0000 UTC" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637726 5011 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15734h13m37.565958621s" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637928 5011 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.637939 5011 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.638009 5011 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.638705 5011 factory.go:55] Registering systemd factory Oct 31 00:18:51.639642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.638721 5011 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640229 5011 factory.go:153] Registering CRI-O factory Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640248 5011 factory.go:223] Registration of the crio container factory successfully Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640302 5011 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640322 5011 factory.go:103] Registering Raw factory Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640338 5011 manager.go:1199] Started watching for new ooms in manager Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640674 5011 manager.go:319] Starting recovery of all containers Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640855 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ef4d6fd-df80-4242-9971-19f9ff9316c5" volumeName="kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640895 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90bc26e1-5d51-46e9-8832-d12588bc7ac3" volumeName="kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640908 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d" volumeName="kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640920 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640931 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b" volumeName="kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640942 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8ed8c6-29da-49fc-962a-a0f2bc51f12a" volumeName="kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640960 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90bc26e1-5d51-46e9-8832-d12588bc7ac3" volumeName="kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640970 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e356e65d-d1de-4fb4-8a77-0707aacf1999" volumeName="kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640984 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494ca01f-586a-4d2b-8625-5e56046ecf86" volumeName="kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.640997 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90bc26e1-5d51-46e9-8832-d12588bc7ac3" volumeName="kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641008 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b22cf76-b18b-4ce7-84e3-0c4940594b5c" volumeName="kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641019 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d" volumeName="kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641031 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d34220d9-9132-4909-a542-35d25dd1af45" volumeName="kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641043 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641057 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641069 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25" volumeName="kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641082 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="171f86cc-abaf-4fec-8ca2-d4809d731a37" volumeName="kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641094 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641104 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee655651-2201-4c07-9b83-10440b2ed2c2" volumeName="kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641116 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee655651-2201-4c07-9b83-10440b2ed2c2" volumeName="kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641126 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641138 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641163 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" volumeName="kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641176 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bbdb593-002d-4500-8f49-9a3208c8388f" volumeName="kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641188 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8ed8c6-29da-49fc-962a-a0f2bc51f12a" volumeName="kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641201 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7dc021f-37cc-43d4-a4e2-50e449ff0032" volumeName="kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641228 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494ca01f-586a-4d2b-8625-5e56046ecf86" volumeName="kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641240 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29dcb073-2ad3-4d21-91e2-24e2536de375" volumeName="kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641269 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d" volumeName="kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641281 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45b345b2-4e5d-4510-8d14-17dacce01a56" volumeName="kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641292 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea8d015-5b9b-4bf4-b7b1-cd26b741a77c" volumeName="kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641304 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5574e12-afad-4304-b163-f30046d15fc1" volumeName="kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641315 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e127939f-dcd5-405f-97c6-87b7a68a5c36" volumeName="kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641330 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="171f86cc-abaf-4fec-8ca2-d4809d731a37" volumeName="kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641343 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1" volumeName="kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641354 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="886edf34-51d0-43c5-84c3-a1996c07aacb" volumeName="kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641367 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea8d015-5b9b-4bf4-b7b1-cd26b741a77c" volumeName="kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641380 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d34220d9-9132-4909-a542-35d25dd1af45" volumeName="kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641392 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e356e65d-d1de-4fb4-8a77-0707aacf1999" volumeName="kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641404 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45a542cb-bea5-4d21-9208-07c3712bda6e" volumeName="kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641415 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a194a3a1-a93d-44ad-a58c-8d5b8edef450" volumeName="kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641429 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641443 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e127939f-dcd5-405f-97c6-87b7a68a5c36" volumeName="kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641457 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b" volumeName="kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641469 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1" volumeName="kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:18:51.641452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641482 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf37dbf-0b0a-4b33-a3a6-07be5096b090" volumeName="kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641496 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5adc70d-63ec-4ab8-9ca5-8fb4928040fc" volumeName="kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641509 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea8d015-5b9b-4bf4-b7b1-cd26b741a77c" volumeName="kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641520 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d34220d9-9132-4909-a542-35d25dd1af45" volumeName="kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641532 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641543 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="193040bf-22d1-4852-acfe-b2b3dd70b1a8" volumeName="kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641555 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29dcb073-2ad3-4d21-91e2-24e2536de375" volumeName="kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641567 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45b345b2-4e5d-4510-8d14-17dacce01a56" volumeName="kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641577 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c491ada7-6f46-4e40-97f0-699cc84dc60d" volumeName="kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641590 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641604 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641629 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641650 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641663 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29dcb073-2ad3-4d21-91e2-24e2536de375" volumeName="kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641676 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90bc26e1-5d51-46e9-8832-d12588bc7ac3" volumeName="kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641688 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d34220d9-9132-4909-a542-35d25dd1af45" volumeName="kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641701 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5574e12-afad-4304-b163-f30046d15fc1" volumeName="kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641713 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641729 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641741 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641754 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" volumeName="kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641768 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="171f86cc-abaf-4fec-8ca2-d4809d731a37" volumeName="kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641779 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="171f86cc-abaf-4fec-8ca2-d4809d731a37" volumeName="kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641794 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bbdb593-002d-4500-8f49-9a3208c8388f" volumeName="kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641808 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5574e12-afad-4304-b163-f30046d15fc1" volumeName="kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641820 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641833 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" volumeName="kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641846 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b" volumeName="kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641859 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5adc70d-63ec-4ab8-9ca5-8fb4928040fc" volumeName="kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641872 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e59fd416-1856-4d1d-a234-93fa7c4575db" volumeName="kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641885 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0db8503-b76b-46c3-8501-b2a9acedd16a" volumeName="kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641897 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641909 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90bc26e1-5d51-46e9-8832-d12588bc7ac3" volumeName="kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641921 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f3cae7-0416-475c-995d-3d906d104d06" volumeName="kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641933 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" volumeName="kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641943 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" volumeName="kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641956 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" volumeName="kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641981 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494ca01f-586a-4d2b-8625-5e56046ecf86" volumeName="kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw" seLinuxMountContext="" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.641994 5011 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:18:51.643824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.642001 5011 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:18:51.665275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.665198 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:51.666362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.666341 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:51.667941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.667887 5011 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:18:51.674626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.674601 5011 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:18:51.674626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.674627 5011 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:18:51.674748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.674646 5011 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:18:51.674748 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.674690 5011 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:18:51.696323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.696282 5011 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:51.701975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.701955 5011 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:18:51.702695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.702672 5011 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:51.702792 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.702718 5011 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:51.702946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.702924 5011 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:51.708549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708523 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="25da4a3578ba1217e4381a7bacfef2558435a8ab1a9e250308dde791855f9a4f" exitCode=0 Oct 31 00:18:51.708648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708551 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="edc1b52528d1aceb89a463291618d547a736559a6b7afe2fbda325b8fce0b629" exitCode=0 Oct 31 00:18:51.708648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708562 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="334b3e266b8d71b1a1e0d60eedf99c1e0b16e71a5766eace8d608bce4b2f5b8f" exitCode=0 Oct 31 00:18:51.708648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708571 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="d4adb56a84feb7f0d489006fbc3d90f4640874125159b87cb15f54bb971c7487" exitCode=0 Oct 31 00:18:51.708648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708578 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="6d36e80c1d5c1aed701c527ee9a38453f5d1100fc01b2233249d8dc8f22c5aea" exitCode=0 Oct 31 00:18:51.708648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.708586 5011 generic.go:356] "Generic (PLEG): container finished" podID="171f86cc-abaf-4fec-8ca2-d4809d731a37" containerID="faee24d0f8163fca2d9dd89b53547699d5d8f2552fa597e86351f3efdec5a360" exitCode=0 Oct 31 00:18:51.726388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.726359 5011 generic.go:356] "Generic (PLEG): container finished" podID="e77ff0c82f82c3043aa6a2e6855afd44" containerID="3451fb5be0ad0da16d40db3a2802c1deb0ea19d82d4d30fbc8e511ce1599f065" exitCode=0 Oct 31 00:18:51.729335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.729315 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:51.729404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.729346 5011 generic.go:356] "Generic (PLEG): container finished" podID="d34220d9-9132-4909-a542-35d25dd1af45" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" exitCode=255 Oct 31 00:18:51.742962 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.742947 5011 manager.go:324] Recovery completed Oct 31 00:18:51.764714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764658 5011 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:18:51.764714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764678 5011 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:18:51.764714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764698 5011 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:18:51.764860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764843 5011 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:18:51.764899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764863 5011 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:18:51.764899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764893 5011 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:18:51.764956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764899 5011 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:18:51.764956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764906 5011 policy_none.go:50] "Start" Oct 31 00:18:51.764956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764915 5011 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:18:51.764956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.764925 5011 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:18:51.765062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.765051 5011 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:18:51.765094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.765063 5011 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:18:51.765094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.765068 5011 policy_none.go:44] "Start" Oct 31 00:18:51.769332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.769307 5011 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:18:51.771789 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.771777 5011 manager.go:341] "Starting Device Plugin manager" Oct 31 00:18:51.772320 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.771802 5011 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:18:51.772320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772321 5011 server.go:87] "Starting device plugin registration server" Oct 31 00:18:51.772558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772548 5011 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:18:51.772607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772558 5011 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:18:51.772668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772648 5011 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:18:51.772793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772754 5011 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:18:51.772848 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772803 5011 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:18:51.772848 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772811 5011 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:18:51.772971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.772889 5011 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:18:51.772792675Z","UUID":"b1114474-679d-4122-9d2d-79a6c05d24a4","Handler":null,"Name":"","Endpoint":""} Oct 31 00:18:51.774452 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.774431 5011 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:18:51.774565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.774480 5011 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:18:51.774565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.774505 5011 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:18:51.774810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.774770 5011 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal"] Oct 31 00:18:51.775939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775641 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.775939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775687 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" event={"ID":"4683e0633b522c7ed8c0620830924e81","Type":"ContainerStarted","Data":"d8f364ee657e518ccd3a73e0c588baa34a3b5e8cfadb5911aa7f16052aade9f3"} Oct 31 00:18:51.775939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775734 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" event={"ID":"4683e0633b522c7ed8c0620830924e81","Type":"ContainerStarted","Data":"e97cd8f797c5584c54ebab49c49054643a63be7d77bf66dc7bee9bf71daf6b2a"} Oct 31 00:18:51.775939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775790 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerStarted","Data":"733b339927e41432fc292de322f26cf6b7ddf3edb344f8e614a884bc47f4484f"} Oct 31 00:18:51.775939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775810 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerDied","Data":"3451fb5be0ad0da16d40db3a2802c1deb0ea19d82d4d30fbc8e511ce1599f065"} Oct 31 00:18:51.776446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.776197 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.776446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.775832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" event={"ID":"e77ff0c82f82c3043aa6a2e6855afd44","Type":"ContainerStarted","Data":"6fc7e6290acf906f3065f665bfde1a411ba8cf1d66570ceae2e33614b7585379"} Oct 31 00:18:51.808162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.808145 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:51.808256 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.808194 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.810339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.810326 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:51.810404 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:51.810355 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.873435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.873413 5011 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:51.874516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.874493 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:51.874585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.874528 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:51.874585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.874542 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-66.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:51.874585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.874579 5011 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.902838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.902821 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.902922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.902876 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.902922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.902893 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.904935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.904919 5011 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:51.904988 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:51.904972 5011 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003153 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003196 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003230 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003161 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003273 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4683e0633b522c7ed8c0620830924e81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-66.ec2.internal\" (UID: \"4683e0633b522c7ed8c0620830924e81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.003407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.003295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77ff0c82f82c3043aa6a2e6855afd44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" (UID: \"e77ff0c82f82c3043aa6a2e6855afd44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.624490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.624457 5011 apiserver.go:52] "Watching apiserver" Oct 31 00:18:52.682123 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.682097 5011 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:18:52.683058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.683034 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw","openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg","openshift-network-console/networking-console-plugin-86c478bb88-pxts4","openshift-insights/periodic-gathering-ts89r-l8pft","kube-system/global-pull-secret-syncer-77tnr","kube-system/kube-apiserver-proxy-ip-10-0-132-66.ec2.internal","openshift-network-diagnostics/network-check-target-q2pbw","openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7","kube-system/konnectivity-agent-klndj","openshift-insights/insights-runtime-extractor-qxclf","kube-system/kas-connection-checker-685895c7c9-q7dwz","openshift-cluster-node-tuning-operator/tuned-gjzn8","kube-system/kas-connection-checker-685895c7c9-msmzl","openshift-console-operator/console-operator-7cd56b8dbc-qb96d","openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd","openshift-network-operator/iptables-alerter-b95dr","openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl","openshift-ovn-kubernetes/ovnkube-node-n26th","openshift-image-registry/image-registry-64c487fb64-crjkj","openshift-multus/multus-m67hq","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr","openshift-image-registry/image-registry-f48cc8d69-w2k7z","openshift-ingress-canary/ingress-canary-scmpl","openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm","openshift-multus/network-metrics-daemon-7mtxv","openshift-image-registry/node-ca-krnph","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5","openshift-dns/dns-default-d476j","openshift-ingress/router-default-55f45b7759-45twl","kube-system/kas-connection-checker-685895c7c9-kw88c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c","openshift-dns/node-resolver-j9cp4","openshift-multus/multus-additional-cni-plugins-rnzgf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal","openshift-insights/insights-operator-7c4986d449-4zwc2"] Oct 31 00:18:52.684893 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.684873 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.688375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.688351 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.689786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.689767 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.690270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.690253 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.690936 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.690920 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:52.693264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.693249 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.694710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.694693 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:52.697368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.697347 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.698558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.698527 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.698787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.698773 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:18:52.699015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.698995 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:18:52.699015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.699008 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g79rw\"" Oct 31 00:18:52.699145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.699133 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q7p8f\"" Oct 31 00:18:52.699602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.699589 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.699696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.699684 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.700565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.700551 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:18:52.700677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.700664 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.700871 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.700857 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.704502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.704480 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:18:52.723481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.723461 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.723647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.723626 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:18:52.723960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.723944 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.724122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724107 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:18:52.724122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724108 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.724259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724231 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:18:52.724259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724241 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.724365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724350 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.724365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724354 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.724436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724369 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:18:52.724629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724615 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:18:52.724751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724740 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:18:52.724795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.724768 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.726435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.726420 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:18:52.726496 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.726445 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.726554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.726542 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.726619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.726608 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ftjzd\"" Oct 31 00:18:52.726714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.726704 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m9w9q\"" Oct 31 00:18:52.727892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.727873 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.727968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.727954 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\"" Oct 31 00:18:52.728020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.727984 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:18:52.728020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.728001 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c22vv\"" Oct 31 00:18:52.728422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.728411 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.728837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.728823 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:18:52.728880 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.728852 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5gqw7\"" Oct 31 00:18:52.729358 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.729345 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.729664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.729650 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.732897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.732879 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:18:52.732960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.732900 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.733519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.733504 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\"" Oct 31 00:18:52.734264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.734249 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.734441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.734424 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hlbxq\"" Oct 31 00:18:52.734656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.734635 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:18:52.734814 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.734802 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-94d5g\"" Oct 31 00:18:52.734978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.734958 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:18:52.735581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.735562 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lwb8l\"" Oct 31 00:18:52.735819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.735806 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.736883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.736868 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:18:52.737032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737016 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:18:52.737075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737021 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:18:52.737075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737064 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737076 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737069 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737101 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nkt5k\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737110 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737120 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737124 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737131 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737112 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737153 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\"" Oct 31 00:18:52.737164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737160 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\"" Oct 31 00:18:52.737458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737182 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\"" Oct 31 00:18:52.737458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737190 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\"" Oct 31 00:18:52.737458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737221 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:18:52.737458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737290 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\"" Oct 31 00:18:52.737458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737326 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\"" Oct 31 00:18:52.737922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737909 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:18:52.737952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737937 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8bqtd\"" Oct 31 00:18:52.737952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.737946 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\"" Oct 31 00:18:52.739344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739332 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.739420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739380 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.739420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739403 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.739473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739432 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:18:52.739473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739446 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:18:52.739533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739521 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.739562 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.739545 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.740166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.740152 5011 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:18:52.742181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742163 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:18:52.742256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742239 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.742303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742253 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.742363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742345 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.742363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742357 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:18:52.742437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742395 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.742437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742397 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.742529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.742484 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.744363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744325 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\"" Oct 31 00:18:52.744363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744359 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:18:52.744513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744374 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:18:52.744513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744376 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744710 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744726 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744746 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744760 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qlrh5\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744768 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744751 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744768 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744769 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744792 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744804 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.744758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744806 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.745303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744818 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:52.745303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.744841 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\"" Oct 31 00:18:52.745466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745438 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745468 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745470 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745481 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745481 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745502 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745521 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\"" Oct 31 00:18:52.745549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745509 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:18:52.745845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.745805 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\"" Oct 31 00:18:52.760491 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.760471 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:52.760553 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.760511 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-66.ec2.internal" Oct 31 00:18:52.806714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806693 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.806794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806718 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.806794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.806794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806748 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.806794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806763 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.806922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806811 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.806922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806839 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:52.806922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806867 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.807103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.806963 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.807103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807004 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.807103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807065 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.807103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807077 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-default-certificate\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.807302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807110 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpwk\" (UniqueName: \"kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:52.807302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807173 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0db8503-b76b-46c3-8501-b2a9acedd16a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:52.807302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807238 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.807302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807240 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.807489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807330 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.807489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.807489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807397 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.807489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807431 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.807489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807448 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807488 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkgh\" (UniqueName: \"kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807513 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-config-volume\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807519 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807561 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807579 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.807707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807623 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g54\" (UniqueName: \"kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.807908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807751 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.808036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808021 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.808093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.807763 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e356e65d-d1de-4fb4-8a77-0707aacf1999-cert\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:52.808093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808075 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.808190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808126 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.808190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808159 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.808304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808225 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.808304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808272 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.808392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808320 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-serving-cert\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.808459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808431 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.808506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808474 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.808548 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.808599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808555 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.808599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808565 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.808689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808631 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqk7\" (UniqueName: \"kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.808689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808670 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.808785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808702 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.808785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808729 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.808785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.808998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.808972 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/494ca01f-586a-4d2b-8625-5e56046ecf86-serving-cert\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.809050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809033 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fkp\" (UniqueName: \"kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp\") pod \"network-check-source-6f78d59f99-fjxrl\" (UID: \"886edf34-51d0-43c5-84c3-a1996c07aacb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:52.809103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809061 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.809256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809235 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.809332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.809438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809422 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-service-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.809518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809502 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-tmp\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.809599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809545 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dt8g\" (UniqueName: \"kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.809735 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809718 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4blr\" (UniqueName: \"kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr\") pod \"kas-connection-checker-685895c7c9-msmzl\" (UID: \"9b22cf76-b18b-4ce7-84e3-0c4940594b5c\") " pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:52.809801 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809759 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.809865 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809852 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.809930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809898 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.809982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809939 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.809982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.809970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.810101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810067 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.810146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810135 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.810229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810189 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.810348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.810413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810365 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.810474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810396 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gb9\" (UniqueName: \"kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9\") pod \"kas-connection-checker-685895c7c9-q7dwz\" (UID: \"d7dc021f-37cc-43d4-a4e2-50e449ff0032\") " pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:52.810609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810575 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.810739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810721 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.810808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810765 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.810869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810827 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.810869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810863 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.810972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810887 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.810972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.810917 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.811091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811001 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.811091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811035 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.811091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811065 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.811264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811092 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.811264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811121 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.811264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811150 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.811264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811180 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.811485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811355 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.811485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811386 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.811485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811418 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hd66\" (UniqueName: \"kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.811485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811451 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811547 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8qj\" (UniqueName: \"kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811575 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811613 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811646 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811683 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811715 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.811765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811747 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811773 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttdf\" (UniqueName: \"kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811802 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.811834 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812028 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812064 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812098 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.812149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812130 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812156 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812237 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgpn\" (UniqueName: \"kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812269 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bzr\" (UniqueName: \"kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr\") pod \"migrator-66df5d496b-sqlxd\" (UID: \"a194a3a1-a93d-44ad-a58c-8d5b8edef450\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812299 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812351 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812416 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-snapshots\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.812462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812459 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812494 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812566 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99nn\" (UniqueName: \"kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812621 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ntw\" (UniqueName: \"kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812655 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812688 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d34220d9-9132-4909-a542-35d25dd1af45-serving-cert\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812691 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812821 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-cni-binary-copy\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812866 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ef4d6fd-df80-4242-9971-19f9ff9316c5-original-pull-secret\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.812895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.813377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.813062 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45b345b2-4e5d-4510-8d14-17dacce01a56-agent-certs\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:52.813377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812754 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcq4h\" (UniqueName: \"kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.813465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.812941 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.813465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.813413 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-trusted-ca\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.813570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.813492 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.813634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.813595 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.813634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.813600 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bbdb593-002d-4500-8f49-9a3208c8388f-data-volume\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.814312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814105 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-tuned\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.814312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814151 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171f86cc-abaf-4fec-8ca2-d4809d731a37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.814312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814151 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-serviceca\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.814312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814268 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e127939f-dcd5-405f-97c6-87b7a68a5c36-tmp-dir\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.814312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814276 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-stats-auth\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814322 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-tmp-dir\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814340 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5574e12-afad-4304-b163-f30046d15fc1-multus-daemon-config\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814462 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814500 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814526 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.814590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814579 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.814835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29dcb073-2ad3-4d21-91e2-24e2536de375-tmp\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.814835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814657 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45b345b2-4e5d-4510-8d14-17dacce01a56-konnectivity-ca\") pod \"konnectivity-agent-klndj\" (UID: \"45b345b2-4e5d-4510-8d14-17dacce01a56\") " pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:52.814835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814696 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tbl\" (UniqueName: \"kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl\") pod \"migrator-66df5d496b-gmllg\" (UID: \"abf37dbf-0b0a-4b33-a3a6-07be5096b090\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:52.814835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814747 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.814960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814883 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qmm\" (UniqueName: \"kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm\") pod \"kas-connection-checker-685895c7c9-kw88c\" (UID: \"193040bf-22d1-4852-acfe-b2b3dd70b1a8\") " pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:52.814960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814914 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.814960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.814938 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:52.815100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815054 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.815100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815083 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.815199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815112 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.815199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815137 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.815199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815145 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-env-overrides\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.815199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815178 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.815199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815224 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xvq\" (UniqueName: \"kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815255 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815282 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815316 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815376 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.815451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815404 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kff94\" (UniqueName: \"kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815491 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815512 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815527 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815542 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815557 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815614 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815638 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-script-lib\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.815698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815670 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815732 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnrk\" (UniqueName: \"kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815740 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815772 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815807 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815834 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815860 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:52.815907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815884 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815909 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815929 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/494ca01f-586a-4d2b-8625-5e56046ecf86-config\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815931 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.815984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvbv\" (UniqueName: \"kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816019 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82nk\" (UniqueName: \"kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816068 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816088 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816106 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.816121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816120 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816129 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovnkube-config\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816145 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816154 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw\") pod \"volume-data-source-validator-58c9f5f46d-hrljw\" (UID: \"c491ada7-6f46-4e40-97f0-699cc84dc60d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816172 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswhd\" (UniqueName: \"kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816187 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816202 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816306 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816431 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816470 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d34220d9-9132-4909-a542-35d25dd1af45-config\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816515 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816562 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhm4r\" (UniqueName: \"kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816581 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816592 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816621 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-iptables-alerter-script\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816633 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.816913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.816874 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90bc26e1-5d51-46e9-8832-d12588bc7ac3-ovn-node-metrics-cert\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.845106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.845086 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g54\" (UniqueName: \"kubernetes.io/projected/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-api-access-96g54\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.845254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.845232 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gb9\" (UniqueName: \"kubernetes.io/projected/d7dc021f-37cc-43d4-a4e2-50e449ff0032-kube-api-access-88gb9\") pod \"kas-connection-checker-685895c7c9-q7dwz\" (UID: \"d7dc021f-37cc-43d4-a4e2-50e449ff0032\") " pod="kube-system/kas-connection-checker-685895c7c9-q7dwz" Oct 31 00:18:52.845348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.845333 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqk7\" (UniqueName: \"kubernetes.io/projected/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-kube-api-access-lhqk7\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.845430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.845418 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fkp\" (UniqueName: \"kubernetes.io/projected/886edf34-51d0-43c5-84c3-a1996c07aacb-kube-api-access-64fkp\") pod \"network-check-source-6f78d59f99-fjxrl\" (UID: \"886edf34-51d0-43c5-84c3-a1996c07aacb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fjxrl" Oct 31 00:18:52.849349 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.849334 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.849605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.849589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkgh\" (UniqueName: \"kubernetes.io/projected/171f86cc-abaf-4fec-8ca2-d4809d731a37-kube-api-access-5kkgh\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.849839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.849821 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4blr\" (UniqueName: \"kubernetes.io/projected/9b22cf76-b18b-4ce7-84e3-0c4940594b5c-kube-api-access-t4blr\") pod \"kas-connection-checker-685895c7c9-msmzl\" (UID: \"9b22cf76-b18b-4ce7-84e3-0c4940594b5c\") " pod="kube-system/kas-connection-checker-685895c7c9-msmzl" Oct 31 00:18:52.852632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.852616 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgpn\" (UniqueName: \"kubernetes.io/projected/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-kube-api-access-5kgpn\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.856036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.856020 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8qj\" (UniqueName: \"kubernetes.io/projected/90bc26e1-5d51-46e9-8832-d12588bc7ac3-kube-api-access-nr8qj\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.856135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.856120 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.858060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.858044 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.858748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.858733 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hd66\" (UniqueName: \"kubernetes.io/projected/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-kube-api-access-8hd66\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.860890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.860872 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttdf\" (UniqueName: \"kubernetes.io/projected/d5574e12-afad-4304-b163-f30046d15fc1-kube-api-access-nttdf\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.860968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.860876 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dt8g\" (UniqueName: \"kubernetes.io/projected/aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d-kube-api-access-2dt8g\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5\" (UID: \"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" Oct 31 00:18:52.861003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.860967 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99nn\" (UniqueName: \"kubernetes.io/projected/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kube-api-access-x99nn\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.863288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.863266 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhm4r\" (UniqueName: \"kubernetes.io/projected/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-kube-api-access-zhm4r\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.866118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.866102 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xvq\" (UniqueName: \"kubernetes.io/projected/ee655651-2201-4c07-9b83-10440b2ed2c2-kube-api-access-b6xvq\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.866243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.866226 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82nk\" (UniqueName: \"kubernetes.io/projected/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-kube-api-access-v82nk\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.867036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.867019 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcq4h\" (UniqueName: \"kubernetes.io/projected/e127939f-dcd5-405f-97c6-87b7a68a5c36-kube-api-access-hcq4h\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.867201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.867186 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/c491ada7-6f46-4e40-97f0-699cc84dc60d-kube-api-access-s6srw\") pod \"volume-data-source-validator-58c9f5f46d-hrljw\" (UID: \"c491ada7-6f46-4e40-97f0-699cc84dc60d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hrljw" Oct 31 00:18:52.867409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.867395 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnrk\" (UniqueName: \"kubernetes.io/projected/29dcb073-2ad3-4d21-91e2-24e2536de375-kube-api-access-gqnrk\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.869095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.869077 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvbv\" (UniqueName: \"kubernetes.io/projected/e356e65d-d1de-4fb4-8a77-0707aacf1999-kube-api-access-xdvbv\") pod \"ingress-canary-scmpl\" (UID: \"e356e65d-d1de-4fb4-8a77-0707aacf1999\") " pod="openshift-ingress-canary/ingress-canary-scmpl" Oct 31 00:18:52.869158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.869128 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kff94\" (UniqueName: \"kubernetes.io/projected/d34220d9-9132-4909-a542-35d25dd1af45-kube-api-access-kff94\") pod \"console-operator-7cd56b8dbc-qb96d\" (UID: \"d34220d9-9132-4909-a542-35d25dd1af45\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:52.869330 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.869315 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.874077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.874062 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"periodic-gathering-ts89r-l8pft\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:18:52.878636 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.878600 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tbl\" (UniqueName: \"kubernetes.io/projected/abf37dbf-0b0a-4b33-a3a6-07be5096b090-kube-api-access-z8tbl\") pod \"migrator-66df5d496b-gmllg\" (UID: \"abf37dbf-0b0a-4b33-a3a6-07be5096b090\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gmllg" Oct 31 00:18:52.879197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.879182 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qmm\" (UniqueName: \"kubernetes.io/projected/193040bf-22d1-4852-acfe-b2b3dd70b1a8-kube-api-access-r6qmm\") pod \"kas-connection-checker-685895c7c9-kw88c\" (UID: \"193040bf-22d1-4852-acfe-b2b3dd70b1a8\") " pod="kube-system/kas-connection-checker-685895c7c9-kw88c" Oct 31 00:18:52.880948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.880933 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rwd\" (UniqueName: \"kubernetes.io/projected/45a542cb-bea5-4d21-9208-07c3712bda6e-kube-api-access-g8rwd\") pod \"network-check-target-q2pbw\" (UID: \"45a542cb-bea5-4d21-9208-07c3712bda6e\") " pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:18:52.883748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.883732 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bzr\" (UniqueName: \"kubernetes.io/projected/a194a3a1-a93d-44ad-a58c-8d5b8edef450-kube-api-access-n6bzr\") pod \"migrator-66df5d496b-sqlxd\" (UID: \"a194a3a1-a93d-44ad-a58c-8d5b8edef450\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sqlxd" Oct 31 00:18:52.883920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.883906 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswhd\" (UniqueName: \"kubernetes.io/projected/f9796d2d-3c0c-4565-a1c3-e8b5b584f97a-kube-api-access-qswhd\") pod \"insights-operator-7c4986d449-4zwc2\" (UID: \"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a\") " pod="openshift-insights/insights-operator-7c4986d449-4zwc2" Oct 31 00:18:52.884092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.884071 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ntw\" (UniqueName: \"kubernetes.io/projected/494ca01f-586a-4d2b-8625-5e56046ecf86-kube-api-access-44ntw\") pod \"service-ca-operator-544d749c8c-cncz7\" (UID: \"494ca01f-586a-4d2b-8625-5e56046ecf86\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" Oct 31 00:18:52.884321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.884306 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpwk\" (UniqueName: \"kubernetes.io/projected/e59fd416-1856-4d1d-a234-93fa7c4575db-kube-api-access-qvpwk\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:52.917876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917853 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:52.917954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.917954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:52.917954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917916 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.917954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917939 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917956 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1-host-slash\") pod \"iptables-alerter-b95dr\" (UID: \"45bfbf31-81b8-4b9b-87ef-86f5c1ad5fa1\") " pod="openshift-network-operator/iptables-alerter-b95dr" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917963 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.917998 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-run\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918019 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-cnibin\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918014 5011 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918090 5011 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-multus-certs\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918134 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.418113108 +0000 UTC m=+1.867200748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : secret "metrics-daemon-secret" not found Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-socket-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918153 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-system-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918159 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls podName:ee655651-2201-4c07-9b83-10440b2ed2c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.418148228 +0000 UTC m=+1.867235850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6nwjr" (UID: "ee655651-2201-4c07-9b83-10440b2ed2c2") : secret "samples-operator-tls" not found Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918178 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918193 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918227 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918251 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918273 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918276 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-netns\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918280 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-systemd\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918305 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-netd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918330 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-cni-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918349 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-conf-dir\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-modprobe-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-sys\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918385 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918402 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918415 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918447 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918463 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918467 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-kubelet-config\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918481 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-sys-fs\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918502 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-multus\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918484 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918519 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-node-log\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918529 5011 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918536 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918521 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5adc70d-63ec-4ab8-9ca5-8fb4928040fc-host\") pod \"node-ca-krnph\" (UID: \"c5adc70d-63ec-4ab8-9ca5-8fb4928040fc\") " pod="openshift-image-registry/node-ca-krnph" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918569 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-cni-bin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918571 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918532 5011 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918587 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.418571773 +0000 UTC m=+1.867659416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-slash\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918616 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918628 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.418614888 +0000 UTC m=+1.867702523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918649 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918666 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918690 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918715 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918731 5011 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918734 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918738 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-kubelet\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918760 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-registration-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.918777 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.418765003 +0000 UTC m=+1.867852637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-etc-selinux\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918716 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-lib-modules\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysconfig\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.918803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918809 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918842 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918860 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918882 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918908 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-host\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918909 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-conf\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918907 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918939 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-kubernetes\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918951 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.918979 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919001 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919026 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-systemd-units\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919048 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919067 5011 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919080 5011 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919078 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-ovn\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919100 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919104 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-cni-bin\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919074 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2bbdb593-002d-4500-8f49-9a3208c8388f-crio-socket\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919123 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-hostroot\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919137 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.419122081 +0000 UTC m=+1.868209713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919149 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e127939f-dcd5-405f-97c6-87b7a68a5c36-hosts-file\") pod \"node-resolver-j9cp4\" (UID: \"e127939f-dcd5-405f-97c6-87b7a68a5c36\") " pod="openshift-dns/node-resolver-j9cp4" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919158 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919183 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919228 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919230 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919231 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-etc-kubernetes\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919250 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-run-systemd\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919254 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919285 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-k8s-cni-cncf-io\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919289 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919327 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919333 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25-device-dir\") pod \"aws-ebs-csi-driver-node-bh99c\" (UID: \"ddf5110e-cdaf-4b6c-9d82-ceefd5b52a25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bh99c" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919355 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919374 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-system-cni-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919383 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919409 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2bbdb593-002d-4500-8f49-9a3208c8388f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919412 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919421 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-multus-socket-dir-parent\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919410 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919454 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919474 5011 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919483 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919512 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.419499287 +0000 UTC m=+1.868586921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919529 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919541 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ef4d6fd-df80-4242-9971-19f9ff9316c5-dbus\") pod \"global-pull-secret-syncer-77tnr\" (UID: \"5ef4d6fd-df80-4242-9971-19f9ff9316c5\") " pod="kube-system/global-pull-secret-syncer-77tnr" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919580 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919534 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-etc-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919596 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919614 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-var-lib-openvswitch\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919616 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919631 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171f86cc-abaf-4fec-8ca2-d4809d731a37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnzgf\" (UID: \"171f86cc-abaf-4fec-8ca2-d4809d731a37\") " pod="openshift-multus/multus-additional-cni-plugins-rnzgf" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919643 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-var-lib-kubelet\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919648 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-host-run-netns\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919660 5011 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919664 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919683 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919705 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.419691414 +0000 UTC m=+1.868779058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919749 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.419740544 +0000 UTC m=+1.868828161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919757 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-etc-sysctl-d\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919759 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dcb073-2ad3-4d21-91e2-24e2536de375-var-lib-kubelet\") pod \"tuned-gjzn8\" (UID: \"29dcb073-2ad3-4d21-91e2-24e2536de375\") " pod="openshift-cluster-node-tuning-operator/tuned-gjzn8" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919784 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919801 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919810 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90bc26e1-5d51-46e9-8832-d12588bc7ac3-log-socket\") pod \"ovnkube-node-n26th\" (UID: \"90bc26e1-5d51-46e9-8832-d12588bc7ac3\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.919849 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5574e12-afad-4304-b163-f30046d15fc1-cnibin\") pod \"multus-m67hq\" (UID: \"d5574e12-afad-4304-b163-f30046d15fc1\") " pod="openshift-multus/multus-m67hq" Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919866 5011 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919877 5011 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:52.920461 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:52.919925 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:53.419913326 +0000 UTC m=+1.869000943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:52.984873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:52.984854 5011 scope.go:122] "RemoveContainer" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" Oct 31 00:18:53.423829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423798 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423846 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423878 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423899 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423922 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.423947 5011 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:53.423977 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.423970 5011 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f48cc8d69-w2k7z: secret "image-registry-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424029 5011 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.423949 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424050 5011 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424064 5011 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c487fb64-crjkj: secret "image-registry-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424053 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls podName:ff76c7b7-ede3-4b47-af5b-ce1562ed50c5 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424007534 +0000 UTC m=+2.873095165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls") pod "image-registry-f48cc8d69-w2k7z" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5") : secret "image-registry-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424072 5011 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424091 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424079796 +0000 UTC m=+2.873167413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424111 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424101692 +0000 UTC m=+2.873189316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424134 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls podName:e5f3cae7-0416-475c-995d-3d906d104d06 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424119146 +0000 UTC m=+2.873206763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls") pod "image-registry-64c487fb64-crjkj" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06") : secret "image-registry-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424145 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls podName:10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424140675 +0000 UTC m=+2.873228293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls") pod "dns-default-d476j" (UID: "10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b") : secret "dns-default-metrics-tls" not found Oct 31 00:18:53.424224 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.424183 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.424235 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.424259 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.424310 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.424329 5011 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424314 5011 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424414 5011 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424362 5011 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424460 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424440186 +0000 UTC m=+2.873527815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : secret "metrics-daemon-secret" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424369 5011 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424482 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert podName:f0db8503-b76b-46c3-8501-b2a9acedd16a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424471385 +0000 UTC m=+2.873559008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxts4" (UID: "f0db8503-b76b-46c3-8501-b2a9acedd16a") : secret "networking-console-plugin-cert" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424499 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424492604 +0000 UTC m=+2.873580226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:53.424660 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.424509 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:54.424503659 +0000 UTC m=+2.873591277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:53.426424 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.426405 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee655651-2201-4c07-9b83-10440b2ed2c2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6nwjr\" (UID: \"ee655651-2201-4c07-9b83-10440b2ed2c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:53.585406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.585381 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" Oct 31 00:18:53.740057 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.739995 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:18:53.740427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.740417 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/1.log" Oct 31 00:18:53.740463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.740447 5011 generic.go:356] "Generic (PLEG): container finished" podID="d34220d9-9132-4909-a542-35d25dd1af45" containerID="ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972" exitCode=255 Oct 31 00:18:53.740546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.740524 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" event={"ID":"d34220d9-9132-4909-a542-35d25dd1af45","Type":"ContainerDied","Data":"ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972"} Oct 31 00:18:53.740591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.740580 5011 scope.go:122] "RemoveContainer" containerID="b24f5c1478659ed96e98ed1ca01e96a80eb9af957f35d9fb274dcdd11a82f12c" Oct 31 00:18:53.740950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.740927 5011 scope.go:122] "RemoveContainer" containerID="ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972" Oct 31 00:18:53.741124 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:53.741101 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:53.889606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.889576 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr"] Oct 31 00:18:53.933984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:53.933969 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:18:54.433301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433271 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433400 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433432 5011 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433497 5011 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433499 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.433481795 +0000 UTC m=+4.882569434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433436 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:54.433564 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433539 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.433527953 +0000 UTC m=+4.882615575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433579 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433615 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.433684 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433763 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.433743148 +0000 UTC m=+4.882830780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433774 5011 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433825 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.433811922 +0000 UTC m=+4.882899554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : secret "router-metrics-certs-default" not found Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433860 5011 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:18:54.433969 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:54.433933 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs podName:e59fd416-1856-4d1d-a234-93fa7c4575db nodeName:}" failed. No retries permitted until 2026-10-31 00:18:56.433918115 +0000 UTC m=+4.883005760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs") pod "network-metrics-daemon-7mtxv" (UID: "e59fd416-1856-4d1d-a234-93fa7c4575db") : secret "metrics-daemon-secret" not found Oct 31 00:18:54.435783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.435757 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b-metrics-tls\") pod \"dns-default-d476j\" (UID: \"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b\") " pod="openshift-dns/dns-default-d476j" Oct 31 00:18:54.435934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.435916 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f0db8503-b76b-46c3-8501-b2a9acedd16a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxts4\" (UID: \"f0db8503-b76b-46c3-8501-b2a9acedd16a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:54.436275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.436256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"image-registry-f48cc8d69-w2k7z\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:54.436319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.436256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"image-registry-64c487fb64-crjkj\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:54.490394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.490373 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:54.491144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.491125 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" Oct 31 00:18:54.494347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.494327 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:18:54.497778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.497761 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d476j" Oct 31 00:18:54.747506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.747435 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:18:54.748931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.748886 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" event={"ID":"ee655651-2201-4c07-9b83-10440b2ed2c2","Type":"ContainerStarted","Data":"b39cef2e42080c9115fe9b42bb3f976b463ad39608debf4691da352ef5d8fdc1"} Oct 31 00:18:54.831690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.831659 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:18:54.843418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.843393 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d476j"] Oct 31 00:18:54.848177 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.848156 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:18:54.849452 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:54.849414 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fec28b_c25d_44f8_a7b2_10e3a4c8cf3b.slice/crio-7659600f2306d911630833351f213e06af4619f59ba32b01346caae45e9fc69a WatchSource:0}: Error finding container 7659600f2306d911630833351f213e06af4619f59ba32b01346caae45e9fc69a: Status 404 returned error can't find the container with id 7659600f2306d911630833351f213e06af4619f59ba32b01346caae45e9fc69a Oct 31 00:18:54.851100 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:54.851076 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f3cae7_0416_475c_995d_3d906d104d06.slice/crio-6a4553cb698c4897c9977947526f61f8e4ab5bbb5968d3f269499069cf4c2c9f WatchSource:0}: Error finding container 6a4553cb698c4897c9977947526f61f8e4ab5bbb5968d3f269499069cf4c2c9f: Status 404 returned error can't find the container with id 6a4553cb698c4897c9977947526f61f8e4ab5bbb5968d3f269499069cf4c2c9f Oct 31 00:18:54.855103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:54.855083 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxts4"] Oct 31 00:18:55.756049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.756010 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d476j" event={"ID":"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b","Type":"ContainerStarted","Data":"7659600f2306d911630833351f213e06af4619f59ba32b01346caae45e9fc69a"} Oct 31 00:18:55.757671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.757640 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" event={"ID":"f0db8503-b76b-46c3-8501-b2a9acedd16a","Type":"ContainerStarted","Data":"da20ea2341dca3a91e72cb8d991990136398aeb4e58c87765a61a54999f563c4"} Oct 31 00:18:55.761876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.761850 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" event={"ID":"e5f3cae7-0416-475c-995d-3d906d104d06","Type":"ContainerStarted","Data":"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a"} Oct 31 00:18:55.761998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.761897 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" event={"ID":"e5f3cae7-0416-475c-995d-3d906d104d06","Type":"ContainerStarted","Data":"6a4553cb698c4897c9977947526f61f8e4ab5bbb5968d3f269499069cf4c2c9f"} Oct 31 00:18:55.763856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.763832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" event={"ID":"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5","Type":"ContainerStarted","Data":"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0"} Oct 31 00:18:55.763944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.763873 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" event={"ID":"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5","Type":"ContainerStarted","Data":"0f9a163b26801705357ec5f52a3bfe9572c30e9bda6486a3a2ae1caf677d73c3"} Oct 31 00:18:55.764021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.763993 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:18:55.770794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.769937 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:55.770794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:55.770300 5011 scope.go:122] "RemoveContainer" containerID="ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972" Oct 31 00:18:55.770794 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:55.770487 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:56.457772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.457731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:18:56.457959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.457789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:56.457959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.457831 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:56.457959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.457875 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:56.457959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.457903 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:18:56.457959 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:56.457913 5011 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:18:56.458196 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:56.457990 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls podName:2bbdb593-002d-4500-8f49-9a3208c8388f nodeName:}" failed. No retries permitted until 2026-10-31 00:19:00.457968555 +0000 UTC m=+8.907056207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qxclf" (UID: "2bbdb593-002d-4500-8f49-9a3208c8388f") : secret "insights-runtime-extractor-tls" not found Oct 31 00:18:56.458196 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:56.458045 5011 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:56.458196 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:56.458090 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls podName:3b8ed8c6-29da-49fc-962a-a0f2bc51f12a nodeName:}" failed. No retries permitted until 2026-10-31 00:19:00.458077195 +0000 UTC m=+8.907164816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-slzmm" (UID: "3b8ed8c6-29da-49fc-962a-a0f2bc51f12a") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:18:56.458196 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:56.458154 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle podName:cea8d015-5b9b-4bf4-b7b1-cd26b741a77c nodeName:}" failed. No retries permitted until 2026-10-31 00:19:00.458143833 +0000 UTC m=+8.907231449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle") pod "router-default-55f45b7759-45twl" (UID: "cea8d015-5b9b-4bf4-b7b1-cd26b741a77c") : configmap references non-existent config key: service-ca.crt Oct 31 00:18:56.460872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.460844 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59fd416-1856-4d1d-a234-93fa7c4575db-metrics-certs\") pod \"network-metrics-daemon-7mtxv\" (UID: \"e59fd416-1856-4d1d-a234-93fa7c4575db\") " pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:56.460993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.460930 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-metrics-certs\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:18:56.595303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.595279 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mtxv" Oct 31 00:18:56.750315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.750289 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-klndj" Oct 31 00:18:56.769505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.769476 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" event={"ID":"ee655651-2201-4c07-9b83-10440b2ed2c2","Type":"ContainerStarted","Data":"435ddaabe8557f5e805f3eeed3b4fa561a23c3479b8bf0867925b803a258811d"} Oct 31 00:18:56.769881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.769536 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6nwjr" event={"ID":"ee655651-2201-4c07-9b83-10440b2ed2c2","Type":"ContainerStarted","Data":"485529ec2104c3f667d43d988dcc0a25325094b2b460234bfb01e8bf7c90495e"} Oct 31 00:18:56.771643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.771613 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d476j" event={"ID":"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b","Type":"ContainerStarted","Data":"057d963e93b3f9aa89a20e0cd907e2c5151197d2c516060876fcc4f27f3df38f"} Oct 31 00:18:56.771737 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.771676 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d476j" event={"ID":"10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b","Type":"ContainerStarted","Data":"a61ea50fd2bf19b497d2cff490fe2a917695fd7d305e415643169f65630a507c"} Oct 31 00:18:56.771737 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.771708 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d476j" Oct 31 00:18:56.773716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.773295 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxts4" event={"ID":"f0db8503-b76b-46c3-8501-b2a9acedd16a","Type":"ContainerStarted","Data":"13b8ce567fb80ac8585d17557ff97327298c32bbbc1106ba9da917a07ad216b3"} Oct 31 00:18:56.773716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.773322 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:56.895867 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:56.895839 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7mtxv"] Oct 31 00:18:56.902075 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:18:56.902050 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59fd416_1856_4d1d_a234_93fa7c4575db.slice/crio-92871cd0790bce9d39f2cf2735ef9a457498ee3684669e87cfa098aa9d295f29 WatchSource:0}: Error finding container 92871cd0790bce9d39f2cf2735ef9a457498ee3684669e87cfa098aa9d295f29: Status 404 returned error can't find the container with id 92871cd0790bce9d39f2cf2735ef9a457498ee3684669e87cfa098aa9d295f29 Oct 31 00:18:57.778359 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:57.778332 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mtxv" event={"ID":"e59fd416-1856-4d1d-a234-93fa7c4575db","Type":"ContainerStarted","Data":"92871cd0790bce9d39f2cf2735ef9a457498ee3684669e87cfa098aa9d295f29"} Oct 31 00:18:57.942581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:57.942541 5011 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:18:57.942926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:57.942913 5011 scope.go:122] "RemoveContainer" containerID="ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972" Oct 31 00:18:57.943095 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:18:57.943079 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qb96d_openshift-console-operator(d34220d9-9132-4909-a542-35d25dd1af45)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" Oct 31 00:18:58.784136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:58.784101 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mtxv" event={"ID":"e59fd416-1856-4d1d-a234-93fa7c4575db","Type":"ContainerStarted","Data":"f37118472e7f4befc99d1c232601870256cdde776b5572cd94846effc8f6d01e"} Oct 31 00:18:58.784510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:18:58.784158 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mtxv" event={"ID":"e59fd416-1856-4d1d-a234-93fa7c4575db","Type":"ContainerStarted","Data":"1dbfd2856d6d2d58dcdef6cd16f207844887fb6a554fd10708621db3c80523c2"} Oct 31 00:19:00.063493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.063463 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:19:00.467012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.466989 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:19:00.492402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.492376 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:19:00.492535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.492433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:19:00.492535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.492455 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:00.493007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.492978 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cea8d015-5b9b-4bf4-b7b1-cd26b741a77c-service-ca-bundle\") pod \"router-default-55f45b7759-45twl\" (UID: \"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c\") " pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:00.494584 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.494564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2bbdb593-002d-4500-8f49-9a3208c8388f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qxclf\" (UID: \"2bbdb593-002d-4500-8f49-9a3208c8388f\") " pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:19:00.494680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.494620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b8ed8c6-29da-49fc-962a-a0f2bc51f12a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-slzmm\" (UID: \"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:19:00.499664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.499646 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qxclf" Oct 31 00:19:00.789299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.789234 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:00.790614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.790595 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" Oct 31 00:19:00.796178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:00.796154 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qxclf"] Oct 31 00:19:00.802116 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:19:00.802085 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbdb593_002d_4500_8f49_9a3208c8388f.slice/crio-e2f557b12a279e41eec3a8f44fa14fd0e64e752729e62ec3f492630a01bc66ed WatchSource:0}: Error finding container e2f557b12a279e41eec3a8f44fa14fd0e64e752729e62ec3f492630a01bc66ed: Status 404 returned error can't find the container with id e2f557b12a279e41eec3a8f44fa14fd0e64e752729e62ec3f492630a01bc66ed Oct 31 00:19:01.103001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.102972 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm"] Oct 31 00:19:01.105921 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:19:01.105896 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8ed8c6_29da_49fc_962a_a0f2bc51f12a.slice/crio-4d99d00108a5f25467ddbc7f018e5fc8dc6a528034986cbb93e8bebe507c8da6 WatchSource:0}: Error finding container 4d99d00108a5f25467ddbc7f018e5fc8dc6a528034986cbb93e8bebe507c8da6: Status 404 returned error can't find the container with id 4d99d00108a5f25467ddbc7f018e5fc8dc6a528034986cbb93e8bebe507c8da6 Oct 31 00:19:01.122815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.122793 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f45b7759-45twl"] Oct 31 00:19:01.125870 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:19:01.125846 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea8d015_5b9b_4bf4_b7b1_cd26b741a77c.slice/crio-f11c1243d7d5b4097ec7c1965f4f0f32176802f3c862ee7037c52aeb2fcdf1b9 WatchSource:0}: Error finding container f11c1243d7d5b4097ec7c1965f4f0f32176802f3c862ee7037c52aeb2fcdf1b9: Status 404 returned error can't find the container with id f11c1243d7d5b4097ec7c1965f4f0f32176802f3c862ee7037c52aeb2fcdf1b9 Oct 31 00:19:01.797007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.796970 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f45b7759-45twl" event={"ID":"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c","Type":"ContainerStarted","Data":"a7047d1297ce6f651461426789c5e4d20168228648dea5cd2eb4649dd375fb1f"} Oct 31 00:19:01.797166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.797029 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f45b7759-45twl" event={"ID":"cea8d015-5b9b-4bf4-b7b1-cd26b741a77c","Type":"ContainerStarted","Data":"f11c1243d7d5b4097ec7c1965f4f0f32176802f3c862ee7037c52aeb2fcdf1b9"} Oct 31 00:19:01.798296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.798263 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" event={"ID":"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a","Type":"ContainerStarted","Data":"4d99d00108a5f25467ddbc7f018e5fc8dc6a528034986cbb93e8bebe507c8da6"} Oct 31 00:19:01.800365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.800341 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qxclf" event={"ID":"2bbdb593-002d-4500-8f49-9a3208c8388f","Type":"ContainerStarted","Data":"c9c92e4c9818c003ef26085e310827baa674ce60f1d39c85fe9ed7d5d5eb5b9d"} Oct 31 00:19:01.800441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.800396 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qxclf" event={"ID":"2bbdb593-002d-4500-8f49-9a3208c8388f","Type":"ContainerStarted","Data":"829b155f4f0d3fb68420d2ef8ae9f7a1f0ffd01e50eb064a06d6e62a0b38a912"} Oct 31 00:19:01.800441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.800416 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qxclf" event={"ID":"2bbdb593-002d-4500-8f49-9a3208c8388f","Type":"ContainerStarted","Data":"e2f557b12a279e41eec3a8f44fa14fd0e64e752729e62ec3f492630a01bc66ed"} Oct 31 00:19:01.830297 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.830273 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:19:01.830487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:01.830472 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:19:02.790152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:02.790127 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:02.793300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:02.793278 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:03.680602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:03.680557 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:19:03.810799 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:03.810764 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-slzmm" event={"ID":"3b8ed8c6-29da-49fc-962a-a0f2bc51f12a","Type":"ContainerStarted","Data":"e3cfb22576946908d8e8cd9e127fc1e19f8679cedf9b29c7f05a123b62d7a71b"} Oct 31 00:19:03.812989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:03.812968 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:19:03.813107 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:03.812997 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qxclf" event={"ID":"2bbdb593-002d-4500-8f49-9a3208c8388f","Type":"ContainerStarted","Data":"266b1b283a8a17c304edd739ca2a2b4a16afd9092b521712734b1bf31f986a7f"} Oct 31 00:19:05.675583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.675548 5011 scope.go:122] "RemoveContainer" containerID="ca1eae42820a63d97121c9be3fd813907870431a97851fe33b038e36105c8972" Oct 31 00:19:05.826278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.826251 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:19:05.826498 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.826370 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" event={"ID":"d34220d9-9132-4909-a542-35d25dd1af45","Type":"ContainerStarted","Data":"1229436e8ac02a814f3ee552e727b3722f592fdbe313d3cfea6cbb772936e67c"} Oct 31 00:19:05.826668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.826649 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:19:05.827627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.827604 5011 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qb96d container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Oct 31 00:19:05.827702 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:05.827643 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" podUID="d34220d9-9132-4909-a542-35d25dd1af45" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Oct 31 00:19:06.780033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:06.780006 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d476j" Oct 31 00:19:06.835042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:06.835011 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qb96d" Oct 31 00:19:08.683102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:08.683063 5011 patch_prober.go:29] interesting pod/image-registry-f48cc8d69-w2k7z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:19:08.683554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:08.683129 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:19:08.901600 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:08.901571 5011 patch_prober.go:29] interesting pod/image-registry-64c487fb64-crjkj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:19:08.901767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:08.901621 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:19:10.233301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:10.233266 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:10.233706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:10.233409 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:19:10.234452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:10.234429 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55f45b7759-45twl" Oct 31 00:19:11.954726 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:11.954694 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:19:11.960119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:11.960087 5011 patch_prober.go:29] interesting pod/image-registry-f48cc8d69-w2k7z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:19:11.960305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:11.960153 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:19:13.685036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:13.684999 5011 patch_prober.go:29] interesting pod/image-registry-64c487fb64-crjkj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:19:13.685414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:13.685053 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:19:20.066039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:20.066005 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q2pbw" Oct 31 00:19:21.847769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.847740 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6lpkq"] Oct 31 00:19:21.882122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.882100 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.897583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.897498 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:19:21.907451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.907428 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pk56l\"" Oct 31 00:19:21.907563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.907486 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:19:21.939937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.939914 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-wtmp\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.939945 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-metrics-client-ca\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.939974 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-sys\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940080 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940150 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940136 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940183 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940161 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtp8f\" (UniqueName: \"kubernetes.io/projected/f24d7ea1-830e-4d93-abb1-21aee7bea13d-kube-api-access-wtp8f\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940181 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-textfile\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940196 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-tls\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.940312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.940261 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-root\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:21.959628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.959609 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:19:21.994536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.994509 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:19:21.994625 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:21.994543 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:19:22.041183 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041149 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-wtmp\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-metrics-client-ca\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041228 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-sys\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041294 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041323 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-sys\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041338 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-wtmp\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtp8f\" (UniqueName: \"kubernetes.io/projected/f24d7ea1-830e-4d93-abb1-21aee7bea13d-kube-api-access-wtp8f\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-textfile\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041419 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-tls\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041464 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-root\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041559 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f24d7ea1-830e-4d93-abb1-21aee7bea13d-root\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.041881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041865 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.042002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.041979 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f24d7ea1-830e-4d93-abb1-21aee7bea13d-metrics-client-ca\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.042124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.042103 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-textfile\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.044026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.043998 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-tls\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.044115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.044055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f24d7ea1-830e-4d93-abb1-21aee7bea13d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.062534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.062501 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtp8f\" (UniqueName: \"kubernetes.io/projected/f24d7ea1-830e-4d93-abb1-21aee7bea13d-kube-api-access-wtp8f\") pod \"node-exporter-6lpkq\" (UID: \"f24d7ea1-830e-4d93-abb1-21aee7bea13d\") " pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.231180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.231105 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lpkq" Oct 31 00:19:22.270308 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:19:22.270282 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24d7ea1_830e_4d93_abb1_21aee7bea13d.slice/crio-65d69b06ce9d01bdecd669a6810b4af97b997723d9622efaafbcd0c7027c8a2e WatchSource:0}: Error finding container 65d69b06ce9d01bdecd669a6810b4af97b997723d9622efaafbcd0c7027c8a2e: Status 404 returned error can't find the container with id 65d69b06ce9d01bdecd669a6810b4af97b997723d9622efaafbcd0c7027c8a2e Oct 31 00:19:22.741193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.741162 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26th" Oct 31 00:19:22.890405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:22.890371 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lpkq" event={"ID":"f24d7ea1-830e-4d93-abb1-21aee7bea13d","Type":"ContainerStarted","Data":"65d69b06ce9d01bdecd669a6810b4af97b997723d9622efaafbcd0c7027c8a2e"} Oct 31 00:19:23.685434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:23.685408 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:19:23.896362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:23.896325 5011 generic.go:356] "Generic (PLEG): container finished" podID="f24d7ea1-830e-4d93-abb1-21aee7bea13d" containerID="e3396a23e676e1dc1b755d23d168a054755da0495ed4ae82a84cf32855c6a7c0" exitCode=0 Oct 31 00:19:23.896746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:23.896408 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lpkq" event={"ID":"f24d7ea1-830e-4d93-abb1-21aee7bea13d","Type":"ContainerDied","Data":"e3396a23e676e1dc1b755d23d168a054755da0495ed4ae82a84cf32855c6a7c0"} Oct 31 00:19:24.902551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:24.902511 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lpkq" event={"ID":"f24d7ea1-830e-4d93-abb1-21aee7bea13d","Type":"ContainerStarted","Data":"0cf3f5655f87e6abe0006adea4ac4c3bd2e6c2fa1eb4c917d6bc5e7fa2074691"} Oct 31 00:19:24.902914 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:24.902562 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lpkq" event={"ID":"f24d7ea1-830e-4d93-abb1-21aee7bea13d","Type":"ContainerStarted","Data":"49dbda092981bcaab726e2aa13fed409e383986493888ab4d5e171bf95efcad9"} Oct 31 00:19:24.980147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:24.980097 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6lpkq" podStartSLOduration=2.6075831320000002 podStartE2EDuration="3.980084755s" totalImagesPullingTime="1.290532871s" totalInitContainerRuntime="81.968752ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:24.969347522 +0000 UTC m=+33.418435163" watchObservedRunningTime="2026-10-31 00:19:24.980084755 +0000 UTC m=+33.429172441" Oct 31 00:19:31.945470 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:19:31.945433 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:19:34.145374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:34.145292 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:19:36.977308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:36.977264 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" containerID="cri-o://4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0" gracePeriod=30 Oct 31 00:19:37.253276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.253248 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:19:37.353745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353711 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.353906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353752 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.353906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353787 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.353906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353835 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.353906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.354142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353906 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.354142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.353960 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.354142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.354008 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") pod \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\" (UID: \"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5\") " Oct 31 00:19:37.354398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.354344 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:37.354398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.354368 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:37.356267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.356246 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:37.356525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.356496 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.356626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.356601 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.356740 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.356721 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:37.356806 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.356787 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "kube-api-access-66qms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:37.360592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.360568 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted" pod "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" (UID: "ff76c7b7-ede3-4b47-af5b-ce1562ed50c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:37.455466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455443 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-trusted-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455466 5011 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455475 5011 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-installation-pull-secrets\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455484 5011 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-registry-certificates\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455493 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66qms\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-kube-api-access-66qms\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455501 5011 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-ca-trust-extracted\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455510 5011 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-image-registry-private-configuration\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.455605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.455519 5011 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5-bound-sa-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:37.950907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.950840 5011 generic.go:356] "Generic (PLEG): container finished" podID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerID="4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0" exitCode=0 Oct 31 00:19:37.950907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.950888 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" event={"ID":"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5","Type":"ContainerDied","Data":"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0"} Oct 31 00:19:37.951066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.950923 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" event={"ID":"ff76c7b7-ede3-4b47-af5b-ce1562ed50c5","Type":"ContainerDied","Data":"0f9a163b26801705357ec5f52a3bfe9572c30e9bda6486a3a2ae1caf677d73c3"} Oct 31 00:19:37.951066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.950941 5011 scope.go:122] "RemoveContainer" containerID="4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0" Oct 31 00:19:37.951066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.950889 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f48cc8d69-w2k7z" Oct 31 00:19:37.958900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.958878 5011 scope.go:122] "RemoveContainer" containerID="4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0" Oct 31 00:19:37.959195 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:19:37.959174 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0\": container with ID starting with 4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0 not found: ID does not exist" containerID="4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0" Oct 31 00:19:37.959290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:37.959228 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0"} err="failed to get container status \"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0\": rpc error: code = NotFound desc = could not find container \"4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0\": container with ID starting with 4e095ecb8990665128ebb859d5f6d16482ddc9b06e10ff62c0f83990a8d867b0 not found: ID does not exist" Oct 31 00:19:38.013466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:38.013442 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:19:38.023945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:38.023921 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f48cc8d69-w2k7z"] Oct 31 00:19:39.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:39.679737 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" path="/var/lib/kubelet/pods/ff76c7b7-ede3-4b47-af5b-ce1562ed50c5/volumes" Oct 31 00:19:57.021151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:57.021118 5011 generic.go:356] "Generic (PLEG): container finished" podID="494ca01f-586a-4d2b-8625-5e56046ecf86" containerID="348098d016f8afe2eded6dc6ecdd4e372c37219e3a94c192baddea37ec1fbe70" exitCode=0 Oct 31 00:19:57.021583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:57.021195 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" event={"ID":"494ca01f-586a-4d2b-8625-5e56046ecf86","Type":"ContainerDied","Data":"348098d016f8afe2eded6dc6ecdd4e372c37219e3a94c192baddea37ec1fbe70"} Oct 31 00:19:57.021583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:57.021576 5011 scope.go:122] "RemoveContainer" containerID="348098d016f8afe2eded6dc6ecdd4e372c37219e3a94c192baddea37ec1fbe70" Oct 31 00:19:58.026944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:58.026910 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cncz7" event={"ID":"494ca01f-586a-4d2b-8625-5e56046ecf86","Type":"ContainerStarted","Data":"230bd196179adb09e32cc1ce38841614b9fc28640adc416ad57b1cd6f3846797"} Oct 31 00:19:59.168871 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.168800 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" containerID="cri-o://0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a" gracePeriod=30 Oct 31 00:19:59.417648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.417619 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:19:59.518526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518493 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518522 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518560 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518592 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518643 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518689 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.518795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518714 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") pod \"e5f3cae7-0416-475c-995d-3d906d104d06\" (UID: \"e5f3cae7-0416-475c-995d-3d906d104d06\") " Oct 31 00:19:59.519045 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.518925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.519416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.519387 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.521254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.521172 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.521414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.521389 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:59.521522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.521466 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.521522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.521497 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:59.521609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.521520 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "kube-api-access-jfqvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:59.525912 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.525893 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted" pod "e5f3cae7-0416-475c-995d-3d906d104d06" (UID: "e5f3cae7-0416-475c-995d-3d906d104d06"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:59.619671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619647 5011 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-registry-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619669 5011 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-installation-pull-secrets\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619681 5011 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-bound-sa-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619691 5011 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-registry-certificates\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619699 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqvb\" (UniqueName: \"kubernetes.io/projected/e5f3cae7-0416-475c-995d-3d906d104d06-kube-api-access-jfqvb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619706 5011 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5f3cae7-0416-475c-995d-3d906d104d06-ca-trust-extracted\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619717 5011 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5f3cae7-0416-475c-995d-3d906d104d06-image-registry-private-configuration\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.619791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:19:59.619727 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3cae7-0416-475c-995d-3d906d104d06-trusted-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:20:00.036065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.036029 5011 generic.go:356] "Generic (PLEG): container finished" podID="e5f3cae7-0416-475c-995d-3d906d104d06" containerID="0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a" exitCode=0 Oct 31 00:20:00.036261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.036073 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" Oct 31 00:20:00.036261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.036117 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" event={"ID":"e5f3cae7-0416-475c-995d-3d906d104d06","Type":"ContainerDied","Data":"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a"} Oct 31 00:20:00.036261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.036171 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c487fb64-crjkj" event={"ID":"e5f3cae7-0416-475c-995d-3d906d104d06","Type":"ContainerDied","Data":"6a4553cb698c4897c9977947526f61f8e4ab5bbb5968d3f269499069cf4c2c9f"} Oct 31 00:20:00.036261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.036190 5011 scope.go:122] "RemoveContainer" containerID="0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a" Oct 31 00:20:00.044196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.044180 5011 scope.go:122] "RemoveContainer" containerID="0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a" Oct 31 00:20:00.044490 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:20:00.044471 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a\": container with ID starting with 0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a not found: ID does not exist" containerID="0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a" Oct 31 00:20:00.044551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.044504 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a"} err="failed to get container status \"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a\": rpc error: code = NotFound desc = could not find container \"0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a\": container with ID starting with 0f3e37b9933685cee3cef364f15b8bd682bc20bba76be7dc364eed80dfdbd80a not found: ID does not exist" Oct 31 00:20:00.094527 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.094504 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:20:00.105171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:00.105141 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64c487fb64-crjkj"] Oct 31 00:20:01.041372 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.041345 5011 generic.go:356] "Generic (PLEG): container finished" podID="f9796d2d-3c0c-4565-a1c3-e8b5b584f97a" containerID="e6b863146d3b9837251cf19ec32be4eb12a19537a098c84b520077597e15e374" exitCode=0 Oct 31 00:20:01.041780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.041423 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" event={"ID":"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a","Type":"ContainerDied","Data":"e6b863146d3b9837251cf19ec32be4eb12a19537a098c84b520077597e15e374"} Oct 31 00:20:01.041780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.041754 5011 scope.go:122] "RemoveContainer" containerID="e6b863146d3b9837251cf19ec32be4eb12a19537a098c84b520077597e15e374" Oct 31 00:20:01.680154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.680121 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" path="/var/lib/kubelet/pods/e5f3cae7-0416-475c-995d-3d906d104d06/volumes" Oct 31 00:20:01.733165 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.733131 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:20:01.733395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.733371 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-ts89r-l8pft" podUID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" containerName="insights-gathering" containerID="cri-o://0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81" gracePeriod=30 Oct 31 00:20:01.949695 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:20:01.949619 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:20:01.965231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965191 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jsrmv-ppst6"] Oct 31 00:20:01.965590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965574 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" Oct 31 00:20:01.965650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965592 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" Oct 31 00:20:01.965650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965623 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" Oct 31 00:20:01.965650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965628 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" Oct 31 00:20:01.965751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965686 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff76c7b7-ede3-4b47-af5b-ce1562ed50c5" containerName="registry" Oct 31 00:20:01.965751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.965696 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f3cae7-0416-475c-995d-3d906d104d06" containerName="registry" Oct 31 00:20:01.968647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.968631 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:01.984834 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.984814 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:20:01.988445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:01.988424 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jsrmv-ppst6"] Oct 31 00:20:02.040179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.040154 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.040302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.040195 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlfg\" (UniqueName: \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.040352 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.040331 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.040387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.040370 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.052093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.052066 5011 generic.go:356] "Generic (PLEG): container finished" podID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" containerID="0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81" exitCode=2 Oct 31 00:20:02.052593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.052115 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ts89r-l8pft" Oct 31 00:20:02.052593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.052137 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ts89r-l8pft" event={"ID":"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a","Type":"ContainerDied","Data":"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81"} Oct 31 00:20:02.052593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.052233 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ts89r-l8pft" event={"ID":"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a","Type":"ContainerDied","Data":"935d5beca79532b36673b5e71bd49610743b05ae9b67142d8f96333f78172a8b"} Oct 31 00:20:02.052593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.052269 5011 scope.go:122] "RemoveContainer" containerID="0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81" Oct 31 00:20:02.054367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.054334 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4zwc2" event={"ID":"f9796d2d-3c0c-4565-a1c3-e8b5b584f97a","Type":"ContainerStarted","Data":"1e23875d45bf56816103e91cde1e5017987df4f2695c8d840fa1ecc2a7c919c4"} Oct 31 00:20:02.061792 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.061763 5011 scope.go:122] "RemoveContainer" containerID="0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81" Oct 31 00:20:02.062054 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:20:02.062034 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81\": container with ID starting with 0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81 not found: ID does not exist" containerID="0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81" Oct 31 00:20:02.062120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.062060 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81"} err="failed to get container status \"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81\": rpc error: code = NotFound desc = could not find container \"0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81\": container with ID starting with 0e5e621f0d3e5f112b51dfdf84d6adeaeb9523ef80f694e93228e47dd8527c81 not found: ID does not exist" Oct 31 00:20:02.141509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141475 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") pod \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " Oct 31 00:20:02.141683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141550 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") pod \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " Oct 31 00:20:02.141683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141597 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") pod \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " Oct 31 00:20:02.141683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") pod \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\" (UID: \"05169c32-bc1b-4f6d-9dc7-500e3bd7f96a\") " Oct 31 00:20:02.141854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.141854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141782 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle" pod "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" (UID: "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:02.141963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.141963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path" pod "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" (UID: "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:02.142065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141965 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.142065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.141973 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle" pod "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" (UID: "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:02.142170 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142033 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlfg\" (UniqueName: \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.142246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142186 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-trusted-ca-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:20:02.142246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142229 5011 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-archives-path\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:20:02.142246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142245 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-service-ca-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:20:02.142428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.142752 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.143006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.142977 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.144606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.144584 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d" pod "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" (UID: "05169c32-bc1b-4f6d-9dc7-500e3bd7f96a"). InnerVolumeSpecName "kube-api-access-w4n8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:02.164243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.164220 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlfg\" (UniqueName: \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\") pod \"periodic-gathering-jsrmv-ppst6\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.242917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.242852 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4n8d\" (UniqueName: \"kubernetes.io/projected/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a-kube-api-access-w4n8d\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:20:02.293303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.293279 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:20:02.404998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.404965 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:20:02.408934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.408909 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-ts89r-l8pft"] Oct 31 00:20:02.592449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:02.592422 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jsrmv-ppst6"] Oct 31 00:20:02.595402 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:20:02.595373 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fc4a8d_d05a_4332_9380_734db52c3fd4.slice/crio-37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207 WatchSource:0}: Error finding container 37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207: Status 404 returned error can't find the container with id 37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207 Oct 31 00:20:03.059267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.059170 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" event={"ID":"b4fc4a8d-d05a-4332-9380-734db52c3fd4","Type":"ContainerStarted","Data":"28bc179dc2e59ac386241680bbc8d89db59e533f7ec522ee25c3806e2008c358"} Oct 31 00:20:03.059267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.059237 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" event={"ID":"b4fc4a8d-d05a-4332-9380-734db52c3fd4","Type":"ContainerStarted","Data":"37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207"} Oct 31 00:20:03.098687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.098633 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" podStartSLOduration=2.098618218 podStartE2EDuration="2.098618218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:03.098050506 +0000 UTC m=+71.547138145" watchObservedRunningTime="2026-10-31 00:20:03.098618218 +0000 UTC m=+71.547705859" Oct 31 00:20:03.113674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.113646 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f45b7759-45twl_cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/router/0.log" Oct 31 00:20:03.226882 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.226850 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scmpl_e356e65d-d1de-4fb4-8a77-0707aacf1999/serve-healthcheck-canary/0.log" Oct 31 00:20:03.680149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:03.680120 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" path="/var/lib/kubelet/pods/05169c32-bc1b-4f6d-9dc7-500e3bd7f96a/volumes" Oct 31 00:20:21.132032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:21.132001 5011 generic.go:356] "Generic (PLEG): container finished" podID="aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d" containerID="b915e683767d31779d772a3454a764ca07295f74a9374f726affd415bd33905c" exitCode=0 Oct 31 00:20:21.132376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:21.132074 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" event={"ID":"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d","Type":"ContainerDied","Data":"b915e683767d31779d772a3454a764ca07295f74a9374f726affd415bd33905c"} Oct 31 00:20:21.132417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:21.132381 5011 scope.go:122] "RemoveContainer" containerID="b915e683767d31779d772a3454a764ca07295f74a9374f726affd415bd33905c" Oct 31 00:20:22.137079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:20:22.137044 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rwnw5" event={"ID":"aa5e6bd8-67a3-4a2c-a636-2ba8a4315a0d","Type":"ContainerStarted","Data":"e284a525c091a3a9ff02b57702675a83ee7a214c05e592f794a4c322607b1b81"} Oct 31 00:20:31.944018 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:20:31.943980 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:21:01.944829 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:21:01.944794 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:21:29.394436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:29.394391 5011 generic.go:356] "Generic (PLEG): container finished" podID="b4fc4a8d-d05a-4332-9380-734db52c3fd4" containerID="28bc179dc2e59ac386241680bbc8d89db59e533f7ec522ee25c3806e2008c358" exitCode=0 Oct 31 00:21:29.394843 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:29.394463 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" event={"ID":"b4fc4a8d-d05a-4332-9380-734db52c3fd4","Type":"ContainerDied","Data":"28bc179dc2e59ac386241680bbc8d89db59e533f7ec522ee25c3806e2008c358"} Oct 31 00:21:30.522007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.521984 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:21:30.667230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667133 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\" (UniqueName: \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\") pod \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " Oct 31 00:21:30.667392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667227 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\") pod \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " Oct 31 00:21:30.667392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667333 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\") pod \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " Oct 31 00:21:30.667392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667368 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\") pod \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\" (UID: \"b4fc4a8d-d05a-4332-9380-734db52c3fd4\") " Oct 31 00:21:30.667594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667572 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle" pod "b4fc4a8d-d05a-4332-9380-734db52c3fd4" (UID: "b4fc4a8d-d05a-4332-9380-734db52c3fd4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:30.667704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667678 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path" pod "b4fc4a8d-d05a-4332-9380-734db52c3fd4" (UID: "b4fc4a8d-d05a-4332-9380-734db52c3fd4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:30.667785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.667769 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle" pod "b4fc4a8d-d05a-4332-9380-734db52c3fd4" (UID: "b4fc4a8d-d05a-4332-9380-734db52c3fd4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:30.669163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.669139 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg" pod "b4fc4a8d-d05a-4332-9380-734db52c3fd4" (UID: "b4fc4a8d-d05a-4332-9380-734db52c3fd4"). InnerVolumeSpecName "kube-api-access-thlfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:30.768110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.768081 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-service-ca-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:21:30.768110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.768110 5011 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4fc4a8d-d05a-4332-9380-734db52c3fd4-archives-path\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:21:30.768110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.768119 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fc4a8d-d05a-4332-9380-734db52c3fd4-trusted-ca-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:21:30.768311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:30.768128 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thlfg\" (UniqueName: \"kubernetes.io/projected/b4fc4a8d-d05a-4332-9380-734db52c3fd4-kube-api-access-thlfg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:21:31.402760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:31.402725 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" event={"ID":"b4fc4a8d-d05a-4332-9380-734db52c3fd4","Type":"ContainerDied","Data":"37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207"} Oct 31 00:21:31.402760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:31.402745 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jsrmv-ppst6" Oct 31 00:21:31.402939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:21:31.402777 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37569dc6aa2bb4ff094d91568a10ae7f5d78cac8f21ea9ce59eefec24ef22207" Oct 31 00:21:31.945108 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:21:31.945075 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:22:01.945192 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:22:01.945154 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:22:07.956263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956228 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5"] Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956518 5011 cpu_manager.go:414] "containerMap: removing container" podUID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" containerName="insights-gathering" Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956528 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" containerName="insights-gathering" Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956539 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b4fc4a8d-d05a-4332-9380-734db52c3fd4" containerName="insights-gathering" Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956544 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fc4a8d-d05a-4332-9380-734db52c3fd4" containerName="insights-gathering" Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956603 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="05169c32-bc1b-4f6d-9dc7-500e3bd7f96a" containerName="insights-gathering" Oct 31 00:22:07.956647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.956610 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fc4a8d-d05a-4332-9380-734db52c3fd4" containerName="insights-gathering" Oct 31 00:22:07.959757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.959736 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:07.983294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.983271 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-k56k5\"" Oct 31 00:22:07.983448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.983291 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:22:07.983448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.983431 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:22:07.990529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:07.990509 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5"] Oct 31 00:22:08.119764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.119733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.119896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.119787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.119896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.119857 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw89\" (UniqueName: \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.221179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.221118 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw89\" (UniqueName: \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.221179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.221168 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.221367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.221199 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.221543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.221528 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.221586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.221568 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.243009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.242987 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw89\" (UniqueName: \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.275282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.275264 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:08.574899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:08.574742 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5"] Oct 31 00:22:08.577724 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:08.577697 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ab1dab_ce33_4c57_9125_3a24555aeffb.slice/crio-920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af WatchSource:0}: Error finding container 920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af: Status 404 returned error can't find the container with id 920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af Oct 31 00:22:09.252801 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.252738 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml"] Oct 31 00:22:09.256371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.256351 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.280796 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.280775 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-k888d\"" Oct 31 00:22:09.282592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.282570 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml"] Oct 31 00:22:09.347445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.347411 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg"] Oct 31 00:22:09.350981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.350961 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.365088 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.365069 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zv64j\"" Oct 31 00:22:09.369733 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.369711 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg"] Oct 31 00:22:09.431375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.431334 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.431522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.431400 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.431522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.431490 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hgj\" (UniqueName: \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.457686 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.457658 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k"] Oct 31 00:22:09.461540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.461517 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.479257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.479235 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gk2dt\"" Oct 31 00:22:09.483430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.483405 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k"] Oct 31 00:22:09.532529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532463 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.532529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532509 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.532715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532631 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.532715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532705 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.532837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532736 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hgj\" (UniqueName: \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.532837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvcd\" (UniqueName: \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.532944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.532884 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.533111 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.533094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.552120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.552093 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" event={"ID":"50ab1dab-ce33-4c57-9125-3a24555aeffb","Type":"ContainerStarted","Data":"920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af"} Oct 31 00:22:09.555468 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.555438 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hgj\" (UniqueName: \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.566607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.566585 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw"] Oct 31 00:22:09.570583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.570567 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.574039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.574020 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:09.583991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.583971 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-nrltf\"" Oct 31 00:22:09.590680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.590656 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw"] Oct 31 00:22:09.633641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633594 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.633641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.633739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633670 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.633739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633695 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.633739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633734 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.633849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.633807 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvcd\" (UniqueName: \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.634650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.634533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.634827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.634805 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.658164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.658140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvcd\" (UniqueName: \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.668764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.668743 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.734893 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.734942 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.734973 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.735014 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.735093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.735134 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdpp\" (UniqueName: \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.735586 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.737231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.735858 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.759385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.759351 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.779230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.779195 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:09.836787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.835608 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.836787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.835674 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdpp\" (UniqueName: \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.836787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.835717 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.836787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.836170 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.836787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.836476 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.859535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.859509 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdpp\" (UniqueName: \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.888879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.888854 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml"] Oct 31 00:22:09.889687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.889661 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:09.893541 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:09.893509 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb56035_79b3_4fc0_877d_ab847646297b.slice/crio-76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71 WatchSource:0}: Error finding container 76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71: Status 404 returned error can't find the container with id 76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71 Oct 31 00:22:09.975689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:09.975661 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg"] Oct 31 00:22:09.979160 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:09.979129 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44526eb_5d4f_420b_8d13_4a128729638c.slice/crio-1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938 WatchSource:0}: Error finding container 1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938: Status 404 returned error can't find the container with id 1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938 Oct 31 00:22:10.093230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.093185 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k"] Oct 31 00:22:10.096378 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:10.096349 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b6410f_158e_4dca_97ae_8fe6f68002ed.slice/crio-25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483 WatchSource:0}: Error finding container 25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483: Status 404 returned error can't find the container with id 25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483 Oct 31 00:22:10.196269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.196240 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw"] Oct 31 00:22:10.200022 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:10.199978 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b137e_5981_4ff0_ad3d_89d2ef29eecd.slice/crio-59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d WatchSource:0}: Error finding container 59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d: Status 404 returned error can't find the container with id 59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d Oct 31 00:22:10.559650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.559552 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" event={"ID":"80b6410f-158e-4dca-97ae-8fe6f68002ed","Type":"ContainerStarted","Data":"25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483"} Oct 31 00:22:10.561265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.561182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" event={"ID":"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd","Type":"ContainerStarted","Data":"59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d"} Oct 31 00:22:10.562870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.562802 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerStarted","Data":"76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71"} Oct 31 00:22:10.566121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.566062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" event={"ID":"b44526eb-5d4f-420b-8d13-4a128729638c","Type":"ContainerStarted","Data":"1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938"} Oct 31 00:22:10.978927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.977327 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst"] Oct 31 00:22:10.981108 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:10.981086 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.001556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.001395 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vtxt8\"" Oct 31 00:22:11.001556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.001527 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:22:11.012601 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.012576 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst"] Oct 31 00:22:11.122197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.122159 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9"] Oct 31 00:22:11.126442 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.125740 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.143127 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.142744 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6476k\"" Oct 31 00:22:11.148233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148156 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebca123-8758-47ed-9812-b22723173e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148405 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp6g\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-kube-api-access-6hp6g\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148480 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148545 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148578 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148608 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebca123-8758-47ed-9812-b22723173e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148633 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.149690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.148658 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.152110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.152035 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9"] Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250122 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250156 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250197 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250244 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250273 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250311 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250346 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250369 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qhz\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-kube-api-access-28qhz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250434 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebca123-8758-47ed-9812-b22723173e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250465 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250493 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250522 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250566 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250606 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebca123-8758-47ed-9812-b22723173e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250637 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.250699 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp6g\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-kube-api-access-6hp6g\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.251468 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.251741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.251986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.253101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.252252 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.254343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.253548 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebca123-8758-47ed-9812-b22723173e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.257850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.257792 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebca123-8758-47ed-9812-b22723173e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.265690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.265663 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebca123-8758-47ed-9812-b22723173e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.275973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.275908 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.296292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.296195 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp6g\" (UniqueName: \"kubernetes.io/projected/cebca123-8758-47ed-9812-b22723173e81-kube-api-access-6hp6g\") pod \"maas-default-gateway-openshift-default-6886b96f9-97mst\" (UID: \"cebca123-8758-47ed-9812-b22723173e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.299993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.299970 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:11.352227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.352161 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.352398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.352270 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354547 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354609 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354639 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354665 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qhz\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-kube-api-access-28qhz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354702 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354740 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.354820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354779 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.355649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.355619 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.355757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.355651 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.355757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.354297 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.355921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.355883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.356316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.356296 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.358609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.358573 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.364938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.364898 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.390463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.390389 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qhz\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-kube-api-access-28qhz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.391633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.391593 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c6e1ee-8686-46d9-b538-ac2c575ea5f9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9\" (UID: \"26c6e1ee-8686-46d9-b538-ac2c575ea5f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.446538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.446487 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:11.640218 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.638528 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst"] Oct 31 00:22:11.643247 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:11.642449 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebca123_8758_47ed_9812_b22723173e81.slice/crio-c3e38e576fef46a772a35ce0ec2303982ccd7030abd1de5e639297010bd435cc WatchSource:0}: Error finding container c3e38e576fef46a772a35ce0ec2303982ccd7030abd1de5e639297010bd435cc: Status 404 returned error can't find the container with id c3e38e576fef46a772a35ce0ec2303982ccd7030abd1de5e639297010bd435cc Oct 31 00:22:11.800008 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:11.799922 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9"] Oct 31 00:22:12.589712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:12.589590 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" event={"ID":"26c6e1ee-8686-46d9-b538-ac2c575ea5f9","Type":"ContainerStarted","Data":"f9bec1306baf00978523a9b67943c50c3210050e0db156737ee2937d6543877f"} Oct 31 00:22:12.592695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:12.592599 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" event={"ID":"cebca123-8758-47ed-9812-b22723173e81","Type":"ContainerStarted","Data":"c3e38e576fef46a772a35ce0ec2303982ccd7030abd1de5e639297010bd435cc"} Oct 31 00:22:17.963824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:17.963787 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:17.964181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:17.963874 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:17.964181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:17.963926 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:18.214259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.213832 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:18.214259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.213902 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:18.214259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.213932 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:18.632891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.632856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" event={"ID":"26c6e1ee-8686-46d9-b538-ac2c575ea5f9","Type":"ContainerStarted","Data":"b50abb883d63b4405154fe730fc2bf27eee33a9135726957e9102a4b50101b74"} Oct 31 00:22:18.637236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.637183 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" event={"ID":"cebca123-8758-47ed-9812-b22723173e81","Type":"ContainerStarted","Data":"8a697d2e205df300de1dd2666ab922ebebdfad0bdd276f06a7214658c8be29ff"} Oct 31 00:22:18.720355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.719478 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" podStartSLOduration=1.315047675 podStartE2EDuration="7.719461867s" totalImagesPullingTime="6.404414192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:18.718427472 +0000 UTC m=+207.167515114" watchObservedRunningTime="2026-10-31 00:22:18.719461867 +0000 UTC m=+207.168549506" Oct 31 00:22:18.766007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:18.765938 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podStartSLOduration=2.449294254 podStartE2EDuration="8.765917829s" totalImagesPullingTime="6.316623575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:18.764345014 +0000 UTC m=+207.213432653" watchObservedRunningTime="2026-10-31 00:22:18.765917829 +0000 UTC m=+207.215005467" Oct 31 00:22:19.301385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.301352 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:19.307047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.307022 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:19.447516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.447437 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:19.453490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.453263 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:19.645258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.645191 5011 generic.go:356] "Generic (PLEG): container finished" podID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerID="48f39203d751eae85ce473cdb9b5483a9780005a6d5289f60a8a72c65b683d0b" exitCode=0 Oct 31 00:22:19.646467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.646407 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" event={"ID":"50ab1dab-ce33-4c57-9125-3a24555aeffb","Type":"ContainerDied","Data":"48f39203d751eae85ce473cdb9b5483a9780005a6d5289f60a8a72c65b683d0b"} Oct 31 00:22:19.646467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.646470 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:19.647083 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.647065 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:19.647161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.647137 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9" Oct 31 00:22:19.648943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:19.648882 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:22:22.666420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.666326 5011 generic.go:356] "Generic (PLEG): container finished" podID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerID="b43565bd3bc2fdbbe7f2fa541a3f236cf67b2a4dc81948acb6f6ef87d91c338e" exitCode=0 Oct 31 00:22:22.666940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.666410 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" event={"ID":"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd","Type":"ContainerDied","Data":"b43565bd3bc2fdbbe7f2fa541a3f236cf67b2a4dc81948acb6f6ef87d91c338e"} Oct 31 00:22:22.668136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.668115 5011 generic.go:356] "Generic (PLEG): container finished" podID="6fb56035-79b3-4fc0-877d-ab847646297b" containerID="ed01d5e8778375f5d992ea5103778ecdea9b3a5998fb05839e31304ef9667de3" exitCode=0 Oct 31 00:22:22.668288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.668141 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerDied","Data":"ed01d5e8778375f5d992ea5103778ecdea9b3a5998fb05839e31304ef9667de3"} Oct 31 00:22:22.669762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.669737 5011 generic.go:356] "Generic (PLEG): container finished" podID="b44526eb-5d4f-420b-8d13-4a128729638c" containerID="c7c167ba20cc7cac8e2f91a00f4ca501d113fef1e4dee0bfbe354f1e474d3e69" exitCode=0 Oct 31 00:22:22.669872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.669763 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" event={"ID":"b44526eb-5d4f-420b-8d13-4a128729638c","Type":"ContainerDied","Data":"c7c167ba20cc7cac8e2f91a00f4ca501d113fef1e4dee0bfbe354f1e474d3e69"} Oct 31 00:22:22.671774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.671753 5011 generic.go:356] "Generic (PLEG): container finished" podID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerID="29abf6120e977822faff84cdb48d61ac32e45bc55614f61c5cc7af6ed2a6893a" exitCode=0 Oct 31 00:22:22.671860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:22.671782 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" event={"ID":"80b6410f-158e-4dca-97ae-8fe6f68002ed","Type":"ContainerDied","Data":"29abf6120e977822faff84cdb48d61ac32e45bc55614f61c5cc7af6ed2a6893a"} Oct 31 00:22:23.677457 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.677381 5011 generic.go:356] "Generic (PLEG): container finished" podID="b44526eb-5d4f-420b-8d13-4a128729638c" containerID="7c5c8df576f2ab22eb6a6d9ebb0a482931bda2d262a0f45a7701482d2041b720" exitCode=0 Oct 31 00:22:23.679456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.679433 5011 generic.go:356] "Generic (PLEG): container finished" podID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerID="43cbf49e543c0545f18a653f0d6a42c9e6087ff260b216df408f5967a5948b89" exitCode=0 Oct 31 00:22:23.679896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.679870 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" event={"ID":"b44526eb-5d4f-420b-8d13-4a128729638c","Type":"ContainerDied","Data":"7c5c8df576f2ab22eb6a6d9ebb0a482931bda2d262a0f45a7701482d2041b720"} Oct 31 00:22:23.679990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.679923 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" event={"ID":"80b6410f-158e-4dca-97ae-8fe6f68002ed","Type":"ContainerDied","Data":"43cbf49e543c0545f18a653f0d6a42c9e6087ff260b216df408f5967a5948b89"} Oct 31 00:22:23.681335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.681295 5011 generic.go:356] "Generic (PLEG): container finished" podID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerID="96029582ab03c166bd96f7f56f82b2c4b281a095747d03d143732d290148e9a3" exitCode=0 Oct 31 00:22:23.681335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:23.681326 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" event={"ID":"50ab1dab-ce33-4c57-9125-3a24555aeffb","Type":"ContainerDied","Data":"96029582ab03c166bd96f7f56f82b2c4b281a095747d03d143732d290148e9a3"} Oct 31 00:22:24.690536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:24.690492 5011 generic.go:356] "Generic (PLEG): container finished" podID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerID="06a47724239286cb93032e0c2db7ba8d23f02fa3d58e05aef9fd99d76e47f51f" exitCode=0 Oct 31 00:22:24.691019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:24.690619 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" event={"ID":"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd","Type":"ContainerDied","Data":"06a47724239286cb93032e0c2db7ba8d23f02fa3d58e05aef9fd99d76e47f51f"} Oct 31 00:22:24.695553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:24.695525 5011 generic.go:356] "Generic (PLEG): container finished" podID="6fb56035-79b3-4fc0-877d-ab847646297b" containerID="b76503bb4cca65b70e389052870faf4b227aa8d89585adef1e2930ae5dbd15dc" exitCode=0 Oct 31 00:22:24.695656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:24.695576 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerDied","Data":"b76503bb4cca65b70e389052870faf4b227aa8d89585adef1e2930ae5dbd15dc"} Oct 31 00:22:31.956042 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:22:31.955966 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:22:35.762903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.762821 5011 generic.go:356] "Generic (PLEG): container finished" podID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerID="9754833e054dd229913c9869bf6142fefe7c36bc27f67a5c48cae91cf5916eaa" exitCode=0 Oct 31 00:22:35.762903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.762881 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" event={"ID":"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd","Type":"ContainerDied","Data":"9754833e054dd229913c9869bf6142fefe7c36bc27f67a5c48cae91cf5916eaa"} Oct 31 00:22:35.764922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.764897 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerStarted","Data":"e9b32dab79d68f69eb81b3d8095cf46819a7424a9cd5db2cd72fab2d0d958d32"} Oct 31 00:22:35.767052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.767032 5011 generic.go:356] "Generic (PLEG): container finished" podID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerID="a285e3120a9ef46cc89a528df4d062e3b3f8a20c63d10dc37d723be7389c1ddc" exitCode=0 Oct 31 00:22:35.767134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.767114 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" event={"ID":"50ab1dab-ce33-4c57-9125-3a24555aeffb","Type":"ContainerDied","Data":"a285e3120a9ef46cc89a528df4d062e3b3f8a20c63d10dc37d723be7389c1ddc"} Oct 31 00:22:35.768902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.768882 5011 generic.go:356] "Generic (PLEG): container finished" podID="b44526eb-5d4f-420b-8d13-4a128729638c" containerID="c412228e3bc762c244f9ba287374a4b37e991fd14be07ab7e40df7267beaebf1" exitCode=0 Oct 31 00:22:35.769009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.768969 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" event={"ID":"b44526eb-5d4f-420b-8d13-4a128729638c","Type":"ContainerDied","Data":"c412228e3bc762c244f9ba287374a4b37e991fd14be07ab7e40df7267beaebf1"} Oct 31 00:22:35.770769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.770749 5011 generic.go:356] "Generic (PLEG): container finished" podID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerID="10e0cea9c3e750b741ce5d82a647ed63f445642d27cd4c6e47fb9be597f9ac31" exitCode=0 Oct 31 00:22:35.770845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:35.770779 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" event={"ID":"80b6410f-158e-4dca-97ae-8fe6f68002ed","Type":"ContainerDied","Data":"10e0cea9c3e750b741ce5d82a647ed63f445642d27cd4c6e47fb9be597f9ac31"} Oct 31 00:22:36.006064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.006020 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" podStartSLOduration=3.306667467 podStartE2EDuration="27.006008092s" totalImagesPullingTime="23.699340625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:35.94351194 +0000 UTC m=+224.392599579" watchObservedRunningTime="2026-10-31 00:22:36.006008092 +0000 UTC m=+224.455095731" Oct 31 00:22:36.776573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.776535 5011 generic.go:356] "Generic (PLEG): container finished" podID="6fb56035-79b3-4fc0-877d-ab847646297b" containerID="e9b32dab79d68f69eb81b3d8095cf46819a7424a9cd5db2cd72fab2d0d958d32" exitCode=0 Oct 31 00:22:36.777019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.776628 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerDied","Data":"e9b32dab79d68f69eb81b3d8095cf46819a7424a9cd5db2cd72fab2d0d958d32"} Oct 31 00:22:36.915047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.915021 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:36.974144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.974119 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:36.999428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.999402 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\") pod \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " Oct 31 00:22:36.999510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.999480 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\" (UniqueName: \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\") pod \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " Oct 31 00:22:36.999561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:36.999508 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\") pod \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\" (UID: \"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd\") " Oct 31 00:22:37.000051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.000024 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle" pod "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" (UID: "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.000274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.000255 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:37.001938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.001909 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp" pod "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" (UID: "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd"). InnerVolumeSpecName "kube-api-access-pqdpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.003731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.003714 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:37.004157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.004137 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util" pod "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" (UID: "2a4b137e-5981-4ff0-ad3d-89d2ef29eecd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.100091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100062 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\") pod \"50ab1dab-ce33-4c57-9125-3a24555aeffb\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " Oct 31 00:22:37.100282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100099 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\") pod \"80b6410f-158e-4dca-97ae-8fe6f68002ed\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " Oct 31 00:22:37.100282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100128 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\") pod \"80b6410f-158e-4dca-97ae-8fe6f68002ed\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " Oct 31 00:22:37.100282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100157 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\") pod \"b44526eb-5d4f-420b-8d13-4a128729638c\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " Oct 31 00:22:37.100282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100183 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\") pod \"b44526eb-5d4f-420b-8d13-4a128729638c\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " Oct 31 00:22:37.100282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100238 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\") pod \"80b6410f-158e-4dca-97ae-8fe6f68002ed\" (UID: \"80b6410f-158e-4dca-97ae-8fe6f68002ed\") " Oct 31 00:22:37.100535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100286 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\") pod \"50ab1dab-ce33-4c57-9125-3a24555aeffb\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " Oct 31 00:22:37.100535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100325 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\" (UniqueName: \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\") pod \"50ab1dab-ce33-4c57-9125-3a24555aeffb\" (UID: \"50ab1dab-ce33-4c57-9125-3a24555aeffb\") " Oct 31 00:22:37.100535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100367 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\" (UniqueName: \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\") pod \"b44526eb-5d4f-420b-8d13-4a128729638c\" (UID: \"b44526eb-5d4f-420b-8d13-4a128729638c\") " Oct 31 00:22:37.100674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100592 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle" pod "50ab1dab-ce33-4c57-9125-3a24555aeffb" (UID: "50ab1dab-ce33-4c57-9125-3a24555aeffb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.100674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100609 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-util\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.100674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100628 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdpp\" (UniqueName: \"kubernetes.io/projected/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-kube-api-access-pqdpp\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.100674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100643 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a4b137e-5981-4ff0-ad3d-89d2ef29eecd-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.100867 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100783 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle" pod "b44526eb-5d4f-420b-8d13-4a128729638c" (UID: "b44526eb-5d4f-420b-8d13-4a128729638c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.100966 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.100724 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle" pod "80b6410f-158e-4dca-97ae-8fe6f68002ed" (UID: "80b6410f-158e-4dca-97ae-8fe6f68002ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.102726 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.102697 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57" pod "80b6410f-158e-4dca-97ae-8fe6f68002ed" (UID: "80b6410f-158e-4dca-97ae-8fe6f68002ed"). InnerVolumeSpecName "kube-api-access-ttw57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.102726 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.102716 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89" pod "50ab1dab-ce33-4c57-9125-3a24555aeffb" (UID: "50ab1dab-ce33-4c57-9125-3a24555aeffb"). InnerVolumeSpecName "kube-api-access-2lw89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.102865 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.102749 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd" pod "b44526eb-5d4f-420b-8d13-4a128729638c" (UID: "b44526eb-5d4f-420b-8d13-4a128729638c"). InnerVolumeSpecName "kube-api-access-lqvcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:37.105028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.105008 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util" pod "b44526eb-5d4f-420b-8d13-4a128729638c" (UID: "b44526eb-5d4f-420b-8d13-4a128729638c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.106694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.106671 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util" pod "80b6410f-158e-4dca-97ae-8fe6f68002ed" (UID: "80b6410f-158e-4dca-97ae-8fe6f68002ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.106932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.106917 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util" pod "50ab1dab-ce33-4c57-9125-3a24555aeffb" (UID: "50ab1dab-ce33-4c57-9125-3a24555aeffb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:37.201203 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201180 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-util\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201225 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lw89\" (UniqueName: \"kubernetes.io/projected/50ab1dab-ce33-4c57-9125-3a24555aeffb-kube-api-access-2lw89\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201243 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvcd\" (UniqueName: \"kubernetes.io/projected/b44526eb-5d4f-420b-8d13-4a128729638c-kube-api-access-lqvcd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201257 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ab1dab-ce33-4c57-9125-3a24555aeffb-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201270 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/80b6410f-158e-4dca-97ae-8fe6f68002ed-kube-api-access-ttw57\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201277 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-util\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201286 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201293 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44526eb-5d4f-420b-8d13-4a128729638c-util\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.201334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.201300 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b6410f-158e-4dca-97ae-8fe6f68002ed-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:37.782298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.782267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" Oct 31 00:22:37.782668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.782273 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644dkkl5" event={"ID":"50ab1dab-ce33-4c57-9125-3a24555aeffb","Type":"ContainerDied","Data":"920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af"} Oct 31 00:22:37.782668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.782436 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920d79a0d40cd5800172d4f8110d9fbbd522f1a97f2461035967c271dbf752af" Oct 31 00:22:37.784509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.784484 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" Oct 31 00:22:37.784612 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.784506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4tjpg" event={"ID":"b44526eb-5d4f-420b-8d13-4a128729638c","Type":"ContainerDied","Data":"1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938"} Oct 31 00:22:37.784612 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.784548 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a03be6be9131b8a5484bcad58d25259038347a166aa49dc09ef846f6dacf938" Oct 31 00:22:37.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.786643 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" Oct 31 00:22:37.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.786638 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cj42k" event={"ID":"80b6410f-158e-4dca-97ae-8fe6f68002ed","Type":"ContainerDied","Data":"25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483"} Oct 31 00:22:37.786771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.786682 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ecca21c686e2f55f6a44f69bd1dc57592d68f1048b531c6bd9aa797c82f483" Oct 31 00:22:37.788739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.788717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" event={"ID":"2a4b137e-5981-4ff0-ad3d-89d2ef29eecd","Type":"ContainerDied","Data":"59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d"} Oct 31 00:22:37.788821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.788751 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f784f4fb3042b3fe00dc77def2b4263d6959197f93e4e99fde6da5f5581b3d" Oct 31 00:22:37.788821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.788813 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7csdmw" Oct 31 00:22:37.920178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:37.920154 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:38.008718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.008653 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\") pod \"6fb56035-79b3-4fc0-877d-ab847646297b\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " Oct 31 00:22:38.008718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.008705 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\") pod \"6fb56035-79b3-4fc0-877d-ab847646297b\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " Oct 31 00:22:38.009016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.008768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\" (UniqueName: \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\") pod \"6fb56035-79b3-4fc0-877d-ab847646297b\" (UID: \"6fb56035-79b3-4fc0-877d-ab847646297b\") " Oct 31 00:22:38.009414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.009387 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle" pod "6fb56035-79b3-4fc0-877d-ab847646297b" (UID: "6fb56035-79b3-4fc0-877d-ab847646297b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:38.010802 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.010778 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj" pod "6fb56035-79b3-4fc0-877d-ab847646297b" (UID: "6fb56035-79b3-4fc0-877d-ab847646297b"). InnerVolumeSpecName "kube-api-access-58hgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:38.014693 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.014664 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util" pod "6fb56035-79b3-4fc0-877d-ab847646297b" (UID: "6fb56035-79b3-4fc0-877d-ab847646297b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:38.110166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.110137 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hgj\" (UniqueName: \"kubernetes.io/projected/6fb56035-79b3-4fc0-877d-ab847646297b-kube-api-access-58hgj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:38.110166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.110163 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-util\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:38.110166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.110174 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fb56035-79b3-4fc0-877d-ab847646297b-bundle\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:22:38.794755 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.794728 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" Oct 31 00:22:38.795132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.794728 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a854ml" event={"ID":"6fb56035-79b3-4fc0-877d-ab847646297b","Type":"ContainerDied","Data":"76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71"} Oct 31 00:22:38.795132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:38.794844 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76f57615bbaeb8f6f8118dfbb6b1f67a58209d0a8a81ac46fe686e4eb140aa71" Oct 31 00:22:39.604117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604080 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds"] Oct 31 00:22:39.604452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604438 5011 cpu_manager.go:414] "containerMap: removing container" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="extract" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604453 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="extract" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604474 5011 cpu_manager.go:414] "containerMap: removing container" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="pull" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604482 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="pull" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604496 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="util" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604501 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="util" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604508 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="extract" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604513 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="extract" Oct 31 00:22:39.604520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604522 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604526 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604536 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604544 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604554 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604561 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604577 5011 cpu_manager.go:414] "containerMap: removing container" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604584 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604593 5011 cpu_manager.go:414] "containerMap: removing container" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604615 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604625 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604632 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604641 5011 cpu_manager.go:414] "containerMap: removing container" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604646 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604653 5011 cpu_manager.go:414] "containerMap: removing container" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604657 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="util" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604663 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604666 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604673 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604677 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="pull" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604686 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604690 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604747 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb56035-79b3-4fc0-877d-ab847646297b" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604755 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b6410f-158e-4dca-97ae-8fe6f68002ed" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604762 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44526eb-5d4f-420b-8d13-4a128729638c" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604767 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ab1dab-ce33-4c57-9125-3a24555aeffb" containerName="extract" Oct 31 00:22:39.604825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.604776 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4b137e-5981-4ff0-ad3d-89d2ef29eecd" containerName="extract" Oct 31 00:22:39.608313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.608292 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.626103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.626079 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:22:39.626246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.626115 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:22:39.626246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.626156 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:22:39.626738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.626716 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:22:39.642836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.642812 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds"] Oct 31 00:22:39.721723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721695 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.721838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.721838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721762 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.721923 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721865 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.721923 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721891 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdz7\" (UniqueName: \"kubernetes.io/projected/88919871-e122-47b0-9961-b592e6e4e2f8-kube-api-access-dqdz7\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.721993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.721920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.730308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.730288 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:22:39.730308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.730288 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2v5hv\"" Oct 31 00:22:39.822885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822856 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822904 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822929 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822956 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822976 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.822991 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdz7\" (UniqueName: \"kubernetes.io/projected/88919871-e122-47b0-9961-b592e6e4e2f8-kube-api-access-dqdz7\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.823568 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.823429 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.825238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.825217 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88919871-e122-47b0-9961-b592e6e4e2f8-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.825478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.825452 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.825832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.825816 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.825979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.825957 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88919871-e122-47b0-9961-b592e6e4e2f8-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.861692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.861637 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdz7\" (UniqueName: \"kubernetes.io/projected/88919871-e122-47b0-9961-b592e6e4e2f8-kube-api-access-dqdz7\") pod \"opendatahub-operator-controller-manager-f465f7bb6-pcgds\" (UID: \"88919871-e122-47b0-9961-b592e6e4e2f8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:39.923623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:39.923596 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:40.315412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:40.315375 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds"] Oct 31 00:22:40.317650 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:40.317623 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88919871_e122_47b0_9961_b592e6e4e2f8.slice/crio-634abfac5230c8182e8566d16617ad0660aeed0fe7b040fbab70663de330a816 WatchSource:0}: Error finding container 634abfac5230c8182e8566d16617ad0660aeed0fe7b040fbab70663de330a816: Status 404 returned error can't find the container with id 634abfac5230c8182e8566d16617ad0660aeed0fe7b040fbab70663de330a816 Oct 31 00:22:40.808182 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:40.808148 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" event={"ID":"88919871-e122-47b0-9961-b592e6e4e2f8","Type":"ContainerStarted","Data":"634abfac5230c8182e8566d16617ad0660aeed0fe7b040fbab70663de330a816"} Oct 31 00:22:43.823522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:43.823490 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" event={"ID":"88919871-e122-47b0-9961-b592e6e4e2f8","Type":"ContainerStarted","Data":"3bea487702d0a408f0204a5ec7312adbef1d15157c0225172a4ec77108255bd4"} Oct 31 00:22:44.829614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:44.829583 5011 generic.go:356] "Generic (PLEG): container finished" podID="88919871-e122-47b0-9961-b592e6e4e2f8" containerID="3bea487702d0a408f0204a5ec7312adbef1d15157c0225172a4ec77108255bd4" exitCode=0 Oct 31 00:22:44.829951 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:44.829664 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" event={"ID":"88919871-e122-47b0-9961-b592e6e4e2f8","Type":"ContainerDied","Data":"3bea487702d0a408f0204a5ec7312adbef1d15157c0225172a4ec77108255bd4"} Oct 31 00:22:45.835968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:45.835930 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" event={"ID":"88919871-e122-47b0-9961-b592e6e4e2f8","Type":"ContainerStarted","Data":"7875c0fbcbad8e022896e932cf896f7f216077ed020fcc800a83845ace78b5bd"} Oct 31 00:22:45.836443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:45.836067 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:45.918764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:45.918717 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" podStartSLOduration=2.879814572 podStartE2EDuration="6.918699889s" totalImagesPullingTime="3.523979748s" totalInitContainerRuntime="514.905569ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:45.907761018 +0000 UTC m=+234.356848659" watchObservedRunningTime="2026-10-31 00:22:45.918699889 +0000 UTC m=+234.367787529" Oct 31 00:22:51.569802 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.569768 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-slrv7"] Oct 31 00:22:51.576431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.576409 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.594932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.594903 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-slrv7"] Oct 31 00:22:51.603672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.603652 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:22:51.603779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.603720 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:22:51.603779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.603729 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d4484\"" Oct 31 00:22:51.714890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.714861 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57j5r\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-kube-api-access-57j5r\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.714890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.714891 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-bound-sa-token\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.816034 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.816006 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57j5r\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-kube-api-access-57j5r\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.816162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.816045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-bound-sa-token\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.838035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.837974 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-bound-sa-token\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.847547 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.847511 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:22:51.867497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.867476 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:22:51.877622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.877603 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57j5r\" (UniqueName: \"kubernetes.io/projected/0f4bc455-3466-4bf2-90e4-02b01bcec23b-kube-api-access-57j5r\") pod \"cert-manager-597b95cb84-slrv7\" (UID: \"0f4bc455-3466-4bf2-90e4-02b01bcec23b\") " pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:51.908604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.908579 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d4484\"" Oct 31 00:22:51.916849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:51.916827 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-slrv7" Oct 31 00:22:52.181256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.181173 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6"] Oct 31 00:22:52.185770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.185751 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:52.209239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.209218 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6"] Oct 31 00:22:52.217637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.217614 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:22:52.217746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.217670 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:22:52.221202 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.221181 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sd2tc\"" Oct 31 00:22:52.227186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.227164 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-slrv7"] Oct 31 00:22:52.229646 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:52.229608 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f4bc455_3466_4bf2_90e4_02b01bcec23b.slice/crio-98c1b9572f460c8dcea1762d547836f3564252ea527bef3156f1340fe6b08e6a WatchSource:0}: Error finding container 98c1b9572f460c8dcea1762d547836f3564252ea527bef3156f1340fe6b08e6a: Status 404 returned error can't find the container with id 98c1b9572f460c8dcea1762d547836f3564252ea527bef3156f1340fe6b08e6a Oct 31 00:22:52.319766 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.319738 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982pf\" (UniqueName: \"kubernetes.io/projected/cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1-kube-api-access-982pf\") pod \"authorino-operator-7b7bb4bc77-qfvp6\" (UID: \"cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:52.421028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.420999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982pf\" (UniqueName: \"kubernetes.io/projected/cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1-kube-api-access-982pf\") pod \"authorino-operator-7b7bb4bc77-qfvp6\" (UID: \"cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:52.448637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.448587 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982pf\" (UniqueName: \"kubernetes.io/projected/cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1-kube-api-access-982pf\") pod \"authorino-operator-7b7bb4bc77-qfvp6\" (UID: \"cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:52.497869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.497846 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:52.809630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.809607 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6"] Oct 31 00:22:52.812368 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:52.812341 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ea1e0_8b4a_429d_8d74_5fbdb32199d1.slice/crio-130b511387b2698b290be6e9609d56558604354b9f685d6fa0150f0113b8f4f9 WatchSource:0}: Error finding container 130b511387b2698b290be6e9609d56558604354b9f685d6fa0150f0113b8f4f9: Status 404 returned error can't find the container with id 130b511387b2698b290be6e9609d56558604354b9f685d6fa0150f0113b8f4f9 Oct 31 00:22:52.868684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.868655 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" event={"ID":"cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1","Type":"ContainerStarted","Data":"130b511387b2698b290be6e9609d56558604354b9f685d6fa0150f0113b8f4f9"} Oct 31 00:22:52.870359 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:52.870324 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-slrv7" event={"ID":"0f4bc455-3466-4bf2-90e4-02b01bcec23b","Type":"ContainerStarted","Data":"98c1b9572f460c8dcea1762d547836f3564252ea527bef3156f1340fe6b08e6a"} Oct 31 00:22:55.888516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:55.888482 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" event={"ID":"cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1","Type":"ContainerStarted","Data":"9ad01b1ddca16a9d2cf854700d9c2f0f87051a22eb869ddf78519e84af4c42e0"} Oct 31 00:22:55.889001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:55.888613 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:22:55.890089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:55.890063 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-slrv7" event={"ID":"0f4bc455-3466-4bf2-90e4-02b01bcec23b","Type":"ContainerStarted","Data":"974d1cf51f3b40f4e05aa446d37ef8ae07dbd3ad97b4984076482e617fb1cf9b"} Oct 31 00:22:55.942072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:55.942022 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" podStartSLOduration=1.313606722 podStartE2EDuration="3.942011122s" totalImagesPullingTime="2.6284044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:55.941420619 +0000 UTC m=+244.390508258" watchObservedRunningTime="2026-10-31 00:22:55.942011122 +0000 UTC m=+244.391098754" Oct 31 00:22:55.985855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:55.985812 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-slrv7" podStartSLOduration=1.772797747 podStartE2EDuration="4.98579737s" totalImagesPullingTime="3.212999623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:55.984344152 +0000 UTC m=+244.433431788" watchObservedRunningTime="2026-10-31 00:22:55.98579737 +0000 UTC m=+244.434885009" Oct 31 00:22:56.834483 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.834448 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8"] Oct 31 00:22:56.837975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.837955 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:22:56.842446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.842410 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-pcgds" Oct 31 00:22:56.865927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.865905 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hdccj\"" Oct 31 00:22:56.866574 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.866553 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8"] Oct 31 00:22:56.962115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:56.962081 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq572\" (UniqueName: \"kubernetes.io/projected/42a1cede-f27c-49c3-ab16-e91935a341ab-kube-api-access-sq572\") pod \"limitador-operator-controller-manager-5778bb4449-k85h8\" (UID: \"42a1cede-f27c-49c3-ab16-e91935a341ab\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:22:57.062847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:57.062814 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq572\" (UniqueName: \"kubernetes.io/projected/42a1cede-f27c-49c3-ab16-e91935a341ab-kube-api-access-sq572\") pod \"limitador-operator-controller-manager-5778bb4449-k85h8\" (UID: \"42a1cede-f27c-49c3-ab16-e91935a341ab\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:22:57.088153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:57.088087 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq572\" (UniqueName: \"kubernetes.io/projected/42a1cede-f27c-49c3-ab16-e91935a341ab-kube-api-access-sq572\") pod \"limitador-operator-controller-manager-5778bb4449-k85h8\" (UID: \"42a1cede-f27c-49c3-ab16-e91935a341ab\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:22:57.150002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:57.149980 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:22:57.454109 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:57.454085 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8"] Oct 31 00:22:57.455787 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:22:57.455758 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a1cede_f27c_49c3_ab16_e91935a341ab.slice/crio-4f2e19a4a54fda2a84ebd19f86b7c67105fce3b0e07967c64e1df0ed2ae73522 WatchSource:0}: Error finding container 4f2e19a4a54fda2a84ebd19f86b7c67105fce3b0e07967c64e1df0ed2ae73522: Status 404 returned error can't find the container with id 4f2e19a4a54fda2a84ebd19f86b7c67105fce3b0e07967c64e1df0ed2ae73522 Oct 31 00:22:57.899556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:57.899523 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" event={"ID":"42a1cede-f27c-49c3-ab16-e91935a341ab","Type":"ContainerStarted","Data":"4f2e19a4a54fda2a84ebd19f86b7c67105fce3b0e07967c64e1df0ed2ae73522"} Oct 31 00:22:59.914901 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:59.914858 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" event={"ID":"42a1cede-f27c-49c3-ab16-e91935a341ab","Type":"ContainerStarted","Data":"fd7381e2a348b6b6cdd6714c8d40008d374ba3de036e61b39ea055970691106d"} Oct 31 00:22:59.915395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:22:59.915008 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:23:01.047124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.047068 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" podStartSLOduration=3.167894934 podStartE2EDuration="5.047053415s" totalImagesPullingTime="1.879158481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:59.996481698 +0000 UTC m=+248.445569336" watchObservedRunningTime="2026-10-31 00:23:01.047053415 +0000 UTC m=+249.496141053" Oct 31 00:23:01.048307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.048285 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql"] Oct 31 00:23:01.051712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.051696 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:01.077593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.077573 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ft677\"" Oct 31 00:23:01.080076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.080052 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql"] Oct 31 00:23:01.084326 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.084306 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 00:23:01.200680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.200656 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvg4t\" (UniqueName: \"kubernetes.io/projected/a139a505-279a-4671-a5f0-62c621eb1b38-kube-api-access-lvg4t\") pod \"dns-operator-controller-manager-8557cbdb84-bfgql\" (UID: \"a139a505-279a-4671-a5f0-62c621eb1b38\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:01.301836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.301767 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvg4t\" (UniqueName: \"kubernetes.io/projected/a139a505-279a-4671-a5f0-62c621eb1b38-kube-api-access-lvg4t\") pod \"dns-operator-controller-manager-8557cbdb84-bfgql\" (UID: \"a139a505-279a-4671-a5f0-62c621eb1b38\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:01.323825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.323806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvg4t\" (UniqueName: \"kubernetes.io/projected/a139a505-279a-4671-a5f0-62c621eb1b38-kube-api-access-lvg4t\") pod \"dns-operator-controller-manager-8557cbdb84-bfgql\" (UID: \"a139a505-279a-4671-a5f0-62c621eb1b38\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:01.363627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.363603 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:01.661710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.661686 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql"] Oct 31 00:23:01.663152 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:23:01.663123 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda139a505_279a_4671_a5f0_62c621eb1b38.slice/crio-fe0285e277e14d56dabb65caf854618a248435621fd1cb44668b2754bd12e5df WatchSource:0}: Error finding container fe0285e277e14d56dabb65caf854618a248435621fd1cb44668b2754bd12e5df: Status 404 returned error can't find the container with id fe0285e277e14d56dabb65caf854618a248435621fd1cb44668b2754bd12e5df Oct 31 00:23:01.924281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:01.924193 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" event={"ID":"a139a505-279a-4671-a5f0-62c621eb1b38","Type":"ContainerStarted","Data":"fe0285e277e14d56dabb65caf854618a248435621fd1cb44668b2754bd12e5df"} Oct 31 00:23:01.944654 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:23:01.944620 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:23:04.940572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:04.940513 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" event={"ID":"a139a505-279a-4671-a5f0-62c621eb1b38","Type":"ContainerStarted","Data":"2459ba63221ceb5eb6e148def14d6f767443013a8b3d0a44d545d93030fd9255"} Oct 31 00:23:04.940963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:04.940649 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:04.987807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:04.987765 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" podStartSLOduration=1.407286306 podStartE2EDuration="3.987752683s" totalImagesPullingTime="2.580466377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:04.986630141 +0000 UTC m=+253.435717779" watchObservedRunningTime="2026-10-31 00:23:04.987752683 +0000 UTC m=+253.436840321" Oct 31 00:23:06.896329 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:06.896300 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qfvp6" Oct 31 00:23:07.617464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.617433 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:23:07.620817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.620801 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.640013 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.639989 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:23:07.648771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.648748 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hbhng\"" Oct 31 00:23:07.753452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.753417 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9l8\" (UniqueName: \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.753452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.753458 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.854542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.854512 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9l8\" (UniqueName: \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.854666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.854554 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.854948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.854926 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.887719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.887664 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9l8\" (UniqueName: \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\") pod \"kuadrant-operator-controller-manager-678d87766f-xpt89\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:07.932575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:07.932549 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:08.267715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:08.267644 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:23:08.281871 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:23:08.281837 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbba5a3_dcc5_4e5a_98f1_822fb9faab2f.slice/crio-9be374eb932f41235659a157e38693d93ff2823c337a7a28ff8689ff8471b632 WatchSource:0}: Error finding container 9be374eb932f41235659a157e38693d93ff2823c337a7a28ff8689ff8471b632: Status 404 returned error can't find the container with id 9be374eb932f41235659a157e38693d93ff2823c337a7a28ff8689ff8471b632 Oct 31 00:23:08.962703 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:08.962668 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" event={"ID":"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f","Type":"ContainerStarted","Data":"9be374eb932f41235659a157e38693d93ff2823c337a7a28ff8689ff8471b632"} Oct 31 00:23:10.921873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:10.921841 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-k85h8" Oct 31 00:23:12.987747 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:12.987708 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" event={"ID":"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f","Type":"ContainerStarted","Data":"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915"} Oct 31 00:23:12.988128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:12.987846 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:13.081260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.081216 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" podStartSLOduration=2.439690132 podStartE2EDuration="6.081191944s" totalImagesPullingTime="3.641501812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:13.069232753 +0000 UTC m=+261.518320386" watchObservedRunningTime="2026-10-31 00:23:13.081191944 +0000 UTC m=+261.530279583" Oct 31 00:23:13.894910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.894876 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms"] Oct 31 00:23:13.898671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.898645 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.907957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.907928 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.907978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908006 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908100 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908217 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908187 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908239 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908292 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908331 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.908353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.908347 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwl4k\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-kube-api-access-bwl4k\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:13.912890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.912869 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4lfzt\"" Oct 31 00:23:13.925671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:13.925649 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms"] Oct 31 00:23:14.009691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009657 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009696 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwl4k\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-kube-api-access-bwl4k\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009767 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009803 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009830 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009855 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009878 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009923 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.010186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.009960 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.011995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.010950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.011995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.010957 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.011995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.011283 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.011995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.011423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.011995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.011819 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.012875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.012855 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.014092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.014072 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.038394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.038372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.038680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.038663 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwl4k\" (UniqueName: \"kubernetes.io/projected/29ad5251-bc5f-4e4d-8163-11929c64c4aa-kube-api-access-bwl4k\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd55n5ms\" (UID: \"29ad5251-bc5f-4e4d-8163-11929c64c4aa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.212027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.211953 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:14.537521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.537496 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms"] Oct 31 00:23:14.539490 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:23:14.539466 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ad5251_bc5f_4e4d_8163_11929c64c4aa.slice/crio-cde0bd57c4a85dca8576deb3ebeebbf909c35ac49eabdd6f99544120ae7a6e90 WatchSource:0}: Error finding container cde0bd57c4a85dca8576deb3ebeebbf909c35ac49eabdd6f99544120ae7a6e90: Status 404 returned error can't find the container with id cde0bd57c4a85dca8576deb3ebeebbf909c35ac49eabdd6f99544120ae7a6e90 Oct 31 00:23:14.541635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.541601 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:14.541730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.541668 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:14.541730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.541701 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:14.998890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.998850 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" event={"ID":"29ad5251-bc5f-4e4d-8163-11929c64c4aa","Type":"ContainerStarted","Data":"a6e03eb3bd5a96bac7a422c7308a6b7594fcce78ba24ab1a948b4d82c669d4f7"} Oct 31 00:23:14.999056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:14.998915 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" event={"ID":"29ad5251-bc5f-4e4d-8163-11929c64c4aa","Type":"ContainerStarted","Data":"cde0bd57c4a85dca8576deb3ebeebbf909c35ac49eabdd6f99544120ae7a6e90"} Oct 31 00:23:15.068355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:15.068305 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" podStartSLOduration=2.068291445 podStartE2EDuration="2.068291445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:15.056281969 +0000 UTC m=+263.505369607" watchObservedRunningTime="2026-10-31 00:23:15.068291445 +0000 UTC m=+263.517379146" Oct 31 00:23:15.212579 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:15.212542 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:15.217545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:15.217522 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:15.949606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:15.949488 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bfgql" Oct 31 00:23:16.003541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:16.003508 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:16.004422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:16.004403 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd55n5ms" Oct 31 00:23:23.995588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:23.995551 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:23:31.945364 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:23:31.945328 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:23:51.706883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:51.706851 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:23:51.707400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:23:51.706964 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:24:01.944867 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:24:01.944829 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:24:16.866614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.866583 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fp76r"] Oct 31 00:24:16.870020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.870003 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:16.886306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.886270 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:24:16.886422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.886335 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:24:16.899815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.899685 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pql5j\"" Oct 31 00:24:16.899984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.899897 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 00:24:16.902824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.902782 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fp76r"] Oct 31 00:24:16.911048 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.911023 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:24:16.914416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.914400 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:16.939599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.939577 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:24:16.945673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.945655 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pclx5\"" Oct 31 00:24:16.947382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.947364 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 00:24:16.979347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.979323 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tzl\" (UniqueName: \"kubernetes.io/projected/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-kube-api-access-x8tzl\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:16.979436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.979359 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-cert\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:16.996082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.996048 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lbvwx"] Oct 31 00:24:16.999498 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:16.999478 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.022406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.022388 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dfjg4\"" Oct 31 00:24:17.024870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.024853 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 00:24:17.026739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.026714 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lbvwx"] Oct 31 00:24:17.041086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.041062 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-cz7rx"] Oct 31 00:24:17.044792 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.044776 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.064732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.064708 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cz7rx"] Oct 31 00:24:17.080184 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080157 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.080311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tzl\" (UniqueName: \"kubernetes.io/projected/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-kube-api-access-x8tzl\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:17.080311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080270 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxsn\" (UniqueName: \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.080311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080304 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-cert\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:17.080440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080392 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-tls-certs\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.080518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.080504 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85b8\" (UniqueName: \"kubernetes.io/projected/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-kube-api-access-q85b8\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.082711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.082691 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-cert\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:17.122354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.122297 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tzl\" (UniqueName: \"kubernetes.io/projected/3f82f539-7ac1-438f-a03d-d48c6f0f28f6-kube-api-access-x8tzl\") pod \"kserve-controller-manager-776cc66d47-fp76r\" (UID: \"3f82f539-7ac1-438f-a03d-d48c6f0f28f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:17.124788 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.124771 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gbdlz\"" Oct 31 00:24:17.124881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.124817 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 00:24:17.124930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.124892 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 00:24:17.181068 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181045 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47zf\" (UniqueName: \"kubernetes.io/projected/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-kube-api-access-h47zf\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.181154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181086 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85b8\" (UniqueName: \"kubernetes.io/projected/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-kube-api-access-q85b8\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.181154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181114 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.181267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181177 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-cert\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.181267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181231 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxsn\" (UniqueName: \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.181349 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181313 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-metrics-certs\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.181393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181348 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-tls-certs\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.181768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.181752 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:17.183492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.183469 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.183709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.183692 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-tls-certs\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.207004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.206982 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxsn\" (UniqueName: \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\") pod \"llmisvc-controller-manager-869d5dcc5d-rc6ph\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.208388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.208365 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85b8\" (UniqueName: \"kubernetes.io/projected/ee72f98b-c3db-47cd-b2ad-492dc4f77bd4-kube-api-access-q85b8\") pod \"model-serving-api-75bd89fdfc-lbvwx\" (UID: \"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4\") " pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.226419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.226391 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:17.282156 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.282118 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47zf\" (UniqueName: \"kubernetes.io/projected/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-kube-api-access-h47zf\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.282311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.282260 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-cert\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.282381 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.282326 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-metrics-certs\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.285900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.285880 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-metrics-certs\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.294418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.286912 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-cert\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.311366 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.311252 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:17.313868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.313845 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47zf\" (UniqueName: \"kubernetes.io/projected/f738e6e1-0018-4c35-a89d-6ba0ca8adf0d-kube-api-access-h47zf\") pod \"odh-model-controller-769875db56-cz7rx\" (UID: \"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d\") " pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.356505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.356473 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:17.480277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.480249 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fp76r"] Oct 31 00:24:17.480665 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:24:17.480644 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f82f539_7ac1_438f_a03d_d48c6f0f28f6.slice/crio-23d71bb8bb70de92158e5395f246aeae29516b13b538e00f4190d10aac70ff5a WatchSource:0}: Error finding container 23d71bb8bb70de92158e5395f246aeae29516b13b538e00f4190d10aac70ff5a: Status 404 returned error can't find the container with id 23d71bb8bb70de92158e5395f246aeae29516b13b538e00f4190d10aac70ff5a Oct 31 00:24:17.482002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.481984 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:24:17.535588 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:24:17.535564 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0139065a_3c2e_40c2_9df6_a7a2cae638e7.slice/crio-94ccc7a8d2d02cfebb60576e6e526a9eacbf268a64af8c7935125ba6048da261 WatchSource:0}: Error finding container 94ccc7a8d2d02cfebb60576e6e526a9eacbf268a64af8c7935125ba6048da261: Status 404 returned error can't find the container with id 94ccc7a8d2d02cfebb60576e6e526a9eacbf268a64af8c7935125ba6048da261 Oct 31 00:24:17.535675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.535606 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:24:17.610163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.610138 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lbvwx"] Oct 31 00:24:17.610896 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:24:17.610872 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72f98b_c3db_47cd_b2ad_492dc4f77bd4.slice/crio-9027e8ed7141363cce7793c4126d3d27d53edc581207fada26e0ef4389a94e50 WatchSource:0}: Error finding container 9027e8ed7141363cce7793c4126d3d27d53edc581207fada26e0ef4389a94e50: Status 404 returned error can't find the container with id 9027e8ed7141363cce7793c4126d3d27d53edc581207fada26e0ef4389a94e50 Oct 31 00:24:17.665695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:17.665669 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cz7rx"] Oct 31 00:24:17.666821 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:24:17.666800 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf738e6e1_0018_4c35_a89d_6ba0ca8adf0d.slice/crio-a973a2a182df08b1f552c4da212d38f22fa2100763f84d75049c510833ab68ab WatchSource:0}: Error finding container a973a2a182df08b1f552c4da212d38f22fa2100763f84d75049c510833ab68ab: Status 404 returned error can't find the container with id a973a2a182df08b1f552c4da212d38f22fa2100763f84d75049c510833ab68ab Oct 31 00:24:18.307150 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:18.307043 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" event={"ID":"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4","Type":"ContainerStarted","Data":"9027e8ed7141363cce7793c4126d3d27d53edc581207fada26e0ef4389a94e50"} Oct 31 00:24:18.311010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:18.310939 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" event={"ID":"3f82f539-7ac1-438f-a03d-d48c6f0f28f6","Type":"ContainerStarted","Data":"23d71bb8bb70de92158e5395f246aeae29516b13b538e00f4190d10aac70ff5a"} Oct 31 00:24:18.313349 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:18.313301 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cz7rx" event={"ID":"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d","Type":"ContainerStarted","Data":"a973a2a182df08b1f552c4da212d38f22fa2100763f84d75049c510833ab68ab"} Oct 31 00:24:18.329550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:18.329508 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" event={"ID":"0139065a-3c2e-40c2-9df6-a7a2cae638e7","Type":"ContainerStarted","Data":"94ccc7a8d2d02cfebb60576e6e526a9eacbf268a64af8c7935125ba6048da261"} Oct 31 00:24:21.352932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:21.351976 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" event={"ID":"ee72f98b-c3db-47cd-b2ad-492dc4f77bd4","Type":"ContainerStarted","Data":"db6da6ddce5aaa0ed16fa7797faac51ab6e36bba777c0fa29342f4255f011dec"} Oct 31 00:24:21.352932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:21.352877 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:21.392265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:21.392216 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" podStartSLOduration=2.223316029 podStartE2EDuration="5.392188322s" totalImagesPullingTime="3.168872293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:21.391445289 +0000 UTC m=+329.840532930" watchObservedRunningTime="2026-10-31 00:24:21.392188322 +0000 UTC m=+329.841275964" Oct 31 00:24:22.358259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.358217 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" event={"ID":"3f82f539-7ac1-438f-a03d-d48c6f0f28f6","Type":"ContainerStarted","Data":"14a52162d77fd9ba85097cc05f9d8f0c38cdfae7680c837bc2c72d23e2d6d32a"} Oct 31 00:24:22.358703 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.358406 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:22.359889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.359861 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cz7rx" event={"ID":"f738e6e1-0018-4c35-a89d-6ba0ca8adf0d","Type":"ContainerStarted","Data":"d69d36d37587c43be16aae3a367d708c10d9991f7eeec1ff2bc2dbdd6ef52666"} Oct 31 00:24:22.360017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.359910 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:22.361340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.361316 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" event={"ID":"0139065a-3c2e-40c2-9df6-a7a2cae638e7","Type":"ContainerStarted","Data":"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533"} Oct 31 00:24:22.398332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.398294 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" podStartSLOduration=2.1792139759999998 podStartE2EDuration="6.39828053s" totalImagesPullingTime="4.219066554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:22.396169409 +0000 UTC m=+330.845257046" watchObservedRunningTime="2026-10-31 00:24:22.39828053 +0000 UTC m=+330.847368172" Oct 31 00:24:22.442824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.442779 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" podStartSLOduration=2.455556973 podStartE2EDuration="6.442767655s" totalImagesPullingTime="3.987210682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:22.439776937 +0000 UTC m=+330.888864584" watchObservedRunningTime="2026-10-31 00:24:22.442767655 +0000 UTC m=+330.891855295" Oct 31 00:24:22.479058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:22.479016 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-cz7rx" podStartSLOduration=2.486068178 podStartE2EDuration="6.479003936s" totalImagesPullingTime="3.992935758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:22.477679954 +0000 UTC m=+330.926767590" watchObservedRunningTime="2026-10-31 00:24:22.479003936 +0000 UTC m=+330.928091574" Oct 31 00:24:23.366401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:23.366370 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:24:31.945595 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:24:31.945563 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:24:33.368073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:33.368045 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-cz7rx" Oct 31 00:24:33.370274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:33.370254 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lbvwx" Oct 31 00:24:53.371422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:53.371393 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-fp76r" Oct 31 00:24:54.373151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:24:54.373119 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:25:01.945099 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:25:01.945065 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:25:31.943916 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:25:31.943881 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:26:01.944814 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:26:01.944782 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:26:31.946100 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:26:31.946065 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:27:01.945696 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:27:01.945659 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:27:31.945060 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:27:31.945024 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:28:01.945233 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:28:01.945180 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:28:31.945840 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:28:31.945802 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:28:51.738290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:28:51.738190 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:28:51.740312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:28:51.740291 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:29:01.948243 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:29:01.948194 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:29:31.949104 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:29:31.949064 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:30:01.947251 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:30:01.947200 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:30:31.951902 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:30:31.951865 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:31:01.945344 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:31:01.945310 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:31:31.945246 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:31:31.945185 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:32:01.944812 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:32:01.944777 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:32:31.945995 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:32:31.945958 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:33:01.944964 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:33:01.944929 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:33:31.945055 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:33:31.945016 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:33:34.116766 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.116734 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-xpt89_bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f/manager/0.log" Oct 31 00:33:34.367598 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.367517 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:33:34.367791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.367750 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" podUID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" containerName="manager" containerID="cri-o://528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915" gracePeriod=10 Oct 31 00:33:34.393359 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.393333 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 00:33:34.396715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.396700 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.433669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.433649 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 00:33:34.458745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.458724 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr24g\" (UniqueName: \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.458846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.458766 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.559685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.559644 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.559827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.559810 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr24g\" (UniqueName: \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.560002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.559982 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.583006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.582976 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr24g\" (UniqueName: \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\") pod \"kuadrant-operator-controller-manager-678d87766f-55xxp\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.621188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.621124 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:33:34.749302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.749271 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:34.761695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.761672 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\") pod \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " Oct 31 00:33:34.761805 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.761734 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\" (UniqueName: \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\") pod \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\" (UID: \"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f\") " Oct 31 00:33:34.762011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.761988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume" pod "bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" (UID: "bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:34.763966 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.763940 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8" pod "bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" (UID: "bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f"). InnerVolumeSpecName "kube-api-access-nj9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:34.863128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.863085 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9l8\" (UniqueName: \"kubernetes.io/projected/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-kube-api-access-nj9l8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:33:34.863128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.863128 5011 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f-extensions-socket-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:33:34.980074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.979998 5011 generic.go:356] "Generic (PLEG): container finished" podID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" containerID="528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915" exitCode=0 Oct 31 00:33:34.980074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.980049 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" event={"ID":"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f","Type":"ContainerDied","Data":"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915"} Oct 31 00:33:34.980074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.980051 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" Oct 31 00:33:34.980333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.980089 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89" event={"ID":"bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f","Type":"ContainerDied","Data":"9be374eb932f41235659a157e38693d93ff2823c337a7a28ff8689ff8471b632"} Oct 31 00:33:34.980333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.980111 5011 scope.go:122] "RemoveContainer" containerID="528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915" Oct 31 00:33:34.988949 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.988933 5011 scope.go:122] "RemoveContainer" containerID="528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915" Oct 31 00:33:34.989244 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:33:34.989222 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915\": container with ID starting with 528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915 not found: ID does not exist" containerID="528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915" Oct 31 00:33:34.989315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:34.989251 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915"} err="failed to get container status \"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915\": rpc error: code = NotFound desc = could not find container \"528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915\": container with ID starting with 528dd55887c6aa54b4eb226356bbcb2208d43a46c92ff0fbcf305e636c5c5915 not found: ID does not exist" Oct 31 00:33:35.034220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.034187 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:33:35.038505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.038485 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xpt89"] Oct 31 00:33:35.058750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.058730 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 00:33:35.060972 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:33:35.060949 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552616f2_aac8_4b53_9eee_05986d8b9db7.slice/crio-fa6af5d8e3409c09dfb2bb293489d995d62778a07ce36c2998c967be7bc993b7 WatchSource:0}: Error finding container fa6af5d8e3409c09dfb2bb293489d995d62778a07ce36c2998c967be7bc993b7: Status 404 returned error can't find the container with id fa6af5d8e3409c09dfb2bb293489d995d62778a07ce36c2998c967be7bc993b7 Oct 31 00:33:35.681420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.681335 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" path="/var/lib/kubelet/pods/bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f/volumes" Oct 31 00:33:35.986061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.985976 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" event={"ID":"552616f2-aac8-4b53-9eee-05986d8b9db7","Type":"ContainerStarted","Data":"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96"} Oct 31 00:33:35.986061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.986030 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" event={"ID":"552616f2-aac8-4b53-9eee-05986d8b9db7","Type":"ContainerStarted","Data":"fa6af5d8e3409c09dfb2bb293489d995d62778a07ce36c2998c967be7bc993b7"} Oct 31 00:33:35.986061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:35.986060 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:36.054052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:36.054005 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" podStartSLOduration=2.053990539 podStartE2EDuration="2.053990539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:36.041563654 +0000 UTC m=+884.490651293" watchObservedRunningTime="2026-10-31 00:33:36.053990539 +0000 UTC m=+884.503078177" Oct 31 00:33:46.994313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:46.994283 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 00:33:51.792027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:51.792002 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:33:51.793932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:51.793910 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:33:53.407072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.407032 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:33:53.407578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.407476 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" containerName="manager" Oct 31 00:33:53.407578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.407492 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" containerName="manager" Oct 31 00:33:53.407578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.407562 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbba5a3-dcc5-4e5a-98f1-822fb9faab2f" containerName="manager" Oct 31 00:33:53.411979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.411959 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.429149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.429128 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:33:53.437172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.437148 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:33:53.497168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.497142 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:33:53.503495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.503471 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dbda9462-c01f-4d5c-81c9-a0546b438abb-config-file\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.503580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.503504 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnl4\" (UniqueName: \"kubernetes.io/projected/dbda9462-c01f-4d5c-81c9-a0546b438abb-kube-api-access-xjnl4\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.522645 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.522623 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d6q45\"" Oct 31 00:33:53.604264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.604239 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dbda9462-c01f-4d5c-81c9-a0546b438abb-config-file\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.604370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.604283 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnl4\" (UniqueName: \"kubernetes.io/projected/dbda9462-c01f-4d5c-81c9-a0546b438abb-kube-api-access-xjnl4\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.604860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.604836 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dbda9462-c01f-4d5c-81c9-a0546b438abb-config-file\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.626484 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.626448 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnl4\" (UniqueName: \"kubernetes.io/projected/dbda9462-c01f-4d5c-81c9-a0546b438abb-kube-api-access-xjnl4\") pod \"limitador-limitador-7c9b87d574-n8796\" (UID: \"dbda9462-c01f-4d5c-81c9-a0546b438abb\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:53.724303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:53.724247 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:54.019794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:54.019749 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:33:54.021673 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:33:54.021642 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbda9462_c01f_4d5c_81c9_a0546b438abb.slice/crio-4cd8695314e310fd5c9c2531938ec4a47a0fe69554c133d9bba28f863729cdbe WatchSource:0}: Error finding container 4cd8695314e310fd5c9c2531938ec4a47a0fe69554c133d9bba28f863729cdbe: Status 404 returned error can't find the container with id 4cd8695314e310fd5c9c2531938ec4a47a0fe69554c133d9bba28f863729cdbe Oct 31 00:33:54.023745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:54.023724 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:33:54.072716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:54.072683 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" event={"ID":"dbda9462-c01f-4d5c-81c9-a0546b438abb","Type":"ContainerStarted","Data":"4cd8695314e310fd5c9c2531938ec4a47a0fe69554c133d9bba28f863729cdbe"} Oct 31 00:33:56.084167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:56.084124 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" event={"ID":"dbda9462-c01f-4d5c-81c9-a0546b438abb","Type":"ContainerStarted","Data":"1afa571758573f919778590a468796b2040201e0e9f2ff284460d066a2b55553"} Oct 31 00:33:56.084554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:56.084185 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:33:56.125420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:33:56.125367 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" podStartSLOduration=1.704187782 podStartE2EDuration="3.125355168s" totalImagesPullingTime="1.421167386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:56.123869172 +0000 UTC m=+904.572956811" watchObservedRunningTime="2026-10-31 00:33:56.125355168 +0000 UTC m=+904.574442807" Oct 31 00:34:01.945491 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:34:01.945455 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:34:07.089975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:34:07.089943 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8796" Oct 31 00:34:31.945354 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:34:31.945313 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:35:01.945704 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:35:01.945616 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:35:08.753651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:08.753612 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:35:18.495399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.495363 5011 generic.go:356] "Generic (PLEG): container finished" podID="cebca123-8758-47ed-9812-b22723173e81" containerID="8a697d2e205df300de1dd2666ab922ebebdfad0bdd276f06a7214658c8be29ff" exitCode=0 Oct 31 00:35:18.495850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.495432 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" event={"ID":"cebca123-8758-47ed-9812-b22723173e81","Type":"ContainerDied","Data":"8a697d2e205df300de1dd2666ab922ebebdfad0bdd276f06a7214658c8be29ff"} Oct 31 00:35:18.495850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.495758 5011 scope.go:122] "RemoveContainer" containerID="8a697d2e205df300de1dd2666ab922ebebdfad0bdd276f06a7214658c8be29ff" Oct 31 00:35:18.496733 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.496706 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:35:18.496805 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.496767 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:35:18.496805 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:18.496798 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:35:19.502687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:19.502654 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" event={"ID":"cebca123-8758-47ed-9812-b22723173e81","Type":"ContainerStarted","Data":"088e411ca8e4d339eb856e461bddc466d2b51d670a9354408f7480c7c378fc23"} Oct 31 00:35:19.647606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:19.647568 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:35:20.300830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:20.300799 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:35:20.302031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:20.302007 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:20.302142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:20.302075 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:21.301490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:21.301454 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:21.301970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:21.301528 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:22.300862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:22.300827 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:22.301041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:22.300895 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:23.301448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:23.301401 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:23.301840 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:23.301465 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:24.301346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:24.301305 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:24.301547 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:24.301365 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:25.305190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:25.305158 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:35:25.306179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:25.306159 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" Oct 31 00:35:27.854346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:27.854312 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p"] Oct 31 00:35:27.858007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:27.857987 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:27.879813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:27.879788 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p"] Oct 31 00:35:27.905649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:27.905624 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac10192-0582-4980-ae86-03b73eb7ce1e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:27.905768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:27.905669 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbp8\" (UniqueName: \"kubernetes.io/projected/1ac10192-0582-4980-ae86-03b73eb7ce1e-kube-api-access-mzbp8\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.006563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.006538 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac10192-0582-4980-ae86-03b73eb7ce1e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.006698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.006581 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbp8\" (UniqueName: \"kubernetes.io/projected/1ac10192-0582-4980-ae86-03b73eb7ce1e-kube-api-access-mzbp8\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.008930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.008907 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ac10192-0582-4980-ae86-03b73eb7ce1e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.028707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.028678 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbp8\" (UniqueName: \"kubernetes.io/projected/1ac10192-0582-4980-ae86-03b73eb7ce1e-kube-api-access-mzbp8\") pod \"llmisvc-controller-manager-6466c554d8-9zr4p\" (UID: \"1ac10192-0582-4980-ae86-03b73eb7ce1e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.170683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.170623 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:28.467502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.467474 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p"] Oct 31 00:35:28.469434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:35:28.469400 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac10192_0582_4980_ae86_03b73eb7ce1e.slice/crio-fd174f6760652bc8b813c7bc46d6a857debbca3215a201b463339aab0ca931ff WatchSource:0}: Error finding container fd174f6760652bc8b813c7bc46d6a857debbca3215a201b463339aab0ca931ff: Status 404 returned error can't find the container with id fd174f6760652bc8b813c7bc46d6a857debbca3215a201b463339aab0ca931ff Oct 31 00:35:28.570254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:28.570224 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" event={"ID":"1ac10192-0582-4980-ae86-03b73eb7ce1e","Type":"ContainerStarted","Data":"fd174f6760652bc8b813c7bc46d6a857debbca3215a201b463339aab0ca931ff"} Oct 31 00:35:29.576768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:29.576730 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" event={"ID":"1ac10192-0582-4980-ae86-03b73eb7ce1e","Type":"ContainerStarted","Data":"d138da1fa03ebf82f02147e2d50a6ebb4ed5eefbfda116e095f28c5f976e2ad9"} Oct 31 00:35:29.577141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:29.576816 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:35:29.626114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:35:29.626067 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" podStartSLOduration=2.37051966 podStartE2EDuration="2.626055199s" totalImagesPullingTime="255.535539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:29.614544397 +0000 UTC m=+998.063632037" watchObservedRunningTime="2026-10-31 00:35:29.626055199 +0000 UTC m=+998.075142837" Oct 31 00:35:31.945195 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:35:31.945161 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:36:00.583036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.583010 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9zr4p" Oct 31 00:36:00.650950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.650921 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:36:00.651136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.651114 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" podUID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" containerName="manager" containerID="cri-o://4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533" gracePeriod=30 Oct 31 00:36:00.902571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.902547 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:36:00.961275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.961244 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\" (UniqueName: \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\") pod \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " Oct 31 00:36:00.961414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.961303 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\" (UniqueName: \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\") pod \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\" (UID: \"0139065a-3c2e-40c2-9df6-a7a2cae638e7\") " Oct 31 00:36:00.963563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.963538 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert" pod "0139065a-3c2e-40c2-9df6-a7a2cae638e7" (UID: "0139065a-3c2e-40c2-9df6-a7a2cae638e7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:00.963563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:00.963555 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn" pod "0139065a-3c2e-40c2-9df6-a7a2cae638e7" (UID: "0139065a-3c2e-40c2-9df6-a7a2cae638e7"). InnerVolumeSpecName "kube-api-access-fpxsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:01.062722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.062683 5011 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0139065a-3c2e-40c2-9df6-a7a2cae638e7-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:36:01.062722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.062723 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxsn\" (UniqueName: \"kubernetes.io/projected/0139065a-3c2e-40c2-9df6-a7a2cae638e7-kube-api-access-fpxsn\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:36:01.735417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.735384 5011 generic.go:356] "Generic (PLEG): container finished" podID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" containerID="4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533" exitCode=0 Oct 31 00:36:01.735753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.735422 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" Oct 31 00:36:01.735753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.735471 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" event={"ID":"0139065a-3c2e-40c2-9df6-a7a2cae638e7","Type":"ContainerDied","Data":"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533"} Oct 31 00:36:01.735753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.735521 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph" event={"ID":"0139065a-3c2e-40c2-9df6-a7a2cae638e7","Type":"ContainerDied","Data":"94ccc7a8d2d02cfebb60576e6e526a9eacbf268a64af8c7935125ba6048da261"} Oct 31 00:36:01.735753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.735548 5011 scope.go:122] "RemoveContainer" containerID="4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533" Oct 31 00:36:01.746519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.744670 5011 scope.go:122] "RemoveContainer" containerID="4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533" Oct 31 00:36:01.746884 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:36:01.746862 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533\": container with ID starting with 4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533 not found: ID does not exist" containerID="4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533" Oct 31 00:36:01.746934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.746892 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533"} err="failed to get container status \"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533\": rpc error: code = NotFound desc = could not find container \"4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533\": container with ID starting with 4f6e71b6b43b135863c725decc14aeb0a505e8d5645ed8579d4a7f344dc29533 not found: ID does not exist" Oct 31 00:36:01.784658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.784632 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:36:01.796446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:01.796421 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rc6ph"] Oct 31 00:36:01.944896 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:36:01.944816 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:36:03.681533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:03.681458 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" path="/var/lib/kubelet/pods/0139065a-3c2e-40c2-9df6-a7a2cae638e7/volumes" Oct 31 00:36:20.857620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:20.857582 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:36:25.450159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:25.450123 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:36:31.613672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:31.613641 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:36:31.947315 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:36:31.947232 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:36:39.911508 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:36:39.911470 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:37:01.944799 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:37:01.944756 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:37:11.866358 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.866327 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 00:37:11.866733 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.866718 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" containerName="manager" Oct 31 00:37:11.866775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.866735 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" containerName="manager" Oct 31 00:37:11.866841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.866832 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0139065a-3c2e-40c2-9df6-a7a2cae638e7" containerName="manager" Oct 31 00:37:11.870049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.870030 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.899145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.899116 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 00:37:11.963513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963478 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963564 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963630 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963749 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963656 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963802 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963786 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963816 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl58g\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963876 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:11.963965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:11.963939 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.064976 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065006 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065033 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065047 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065254 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl58g\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065299 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065336 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065368 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065519 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065544 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.065983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.065787 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.066239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.066202 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.067409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.067391 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.067551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.067532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.087937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.087917 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.088159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.088135 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl58g\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\") pod \"maas-default-gateway-openshift-default-6886b96f9-c7jth\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.183710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.183641 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:12.495540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.495517 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 00:37:12.496958 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:37:12.496932 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df53695_6e13_49d5_bc1f_f8fe786df192.slice/crio-18e7d112f8ae1c6c2ef4ee86f8d0dbea6c7ab60d1a5a8211e9209dc52a06c817 WatchSource:0}: Error finding container 18e7d112f8ae1c6c2ef4ee86f8d0dbea6c7ab60d1a5a8211e9209dc52a06c817: Status 404 returned error can't find the container with id 18e7d112f8ae1c6c2ef4ee86f8d0dbea6c7ab60d1a5a8211e9209dc52a06c817 Oct 31 00:37:12.499289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.499255 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:12.499373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.499322 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:12.499373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:12.499351 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:13.113601 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.113570 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" event={"ID":"9df53695-6e13-49d5-bc1f-f8fe786df192","Type":"ContainerStarted","Data":"e115177fbaabc9ff21cc80fd9a9ec7c6c0d8c5800766203b9b761882eb89eaed"} Oct 31 00:37:13.113976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.113622 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" event={"ID":"9df53695-6e13-49d5-bc1f-f8fe786df192","Type":"ContainerStarted","Data":"18e7d112f8ae1c6c2ef4ee86f8d0dbea6c7ab60d1a5a8211e9209dc52a06c817"} Oct 31 00:37:13.184161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.184126 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:13.184501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.184456 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podStartSLOduration=2.184442235 podStartE2EDuration="2.184442235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:13.182722481 +0000 UTC m=+1101.631810122" watchObservedRunningTime="2026-10-31 00:37:13.184442235 +0000 UTC m=+1101.633529868" Oct 31 00:37:13.185981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.185916 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:13.186107 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:13.186020 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:14.184334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:14.184297 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:14.184727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:14.184356 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:15.184241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:15.184186 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:15.184419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:15.184268 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:16.184958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:16.184923 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:16.185366 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:16.184985 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:17.183975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:17.183940 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:17.184177 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:17.183998 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:18.184634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:18.184599 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:18.185123 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:18.184648 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:19.184406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:19.184365 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:37:19.184576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:19.184432 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 00:37:20.617152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:20.617122 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:20.617647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:20.617549 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:20.618413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:20.618390 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 00:37:26.834706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.834669 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 00:37:26.838571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.838554 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.861284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.861263 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 00:37:26.911638 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911611 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911678 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911748 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911824 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911855 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vbl\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.911911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.911877 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.912065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.912012 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:26.912065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:26.912046 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013262 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013290 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vbl\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013317 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013362 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013381 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013405 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013430 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013468 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013773 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013793 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.013932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.013863 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.014057 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.014039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.014200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.014181 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.016115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.016091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.016366 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.016349 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.035436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.035415 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vbl\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.035543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.035486 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vlrk2\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.151249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.151177 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:27.463674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.463645 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 00:37:27.464328 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:37:27.464291 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcb33eb_927a_447d_9a30_cd024560647f.slice/crio-40bcf1ee2b2ad87efee98708ce61b6531e82462bcb16bca52c4be4dce605cce4 WatchSource:0}: Error finding container 40bcf1ee2b2ad87efee98708ce61b6531e82462bcb16bca52c4be4dce605cce4: Status 404 returned error can't find the container with id 40bcf1ee2b2ad87efee98708ce61b6531e82462bcb16bca52c4be4dce605cce4 Oct 31 00:37:27.466397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.466364 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:27.466500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.466428 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:27.466500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:27.466465 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:37:28.217507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:28.217465 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" event={"ID":"bfcb33eb-927a-447d-9a30-cd024560647f","Type":"ContainerStarted","Data":"bf3a271d0867029ba596f6edbc46813245f75b4747ba3e43cbdea2437ef15070"} Oct 31 00:37:28.218079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:28.218054 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" event={"ID":"bfcb33eb-927a-447d-9a30-cd024560647f","Type":"ContainerStarted","Data":"40bcf1ee2b2ad87efee98708ce61b6531e82462bcb16bca52c4be4dce605cce4"} Oct 31 00:37:28.286499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:28.286438 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podStartSLOduration=2.286423261 podStartE2EDuration="2.286423261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:28.273723416 +0000 UTC m=+1116.722811054" watchObservedRunningTime="2026-10-31 00:37:28.286423261 +0000 UTC m=+1116.735510900" Oct 31 00:37:29.152122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:29.152082 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:29.153507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:29.153480 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:29.153614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:29.153531 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:30.151848 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:30.151814 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:30.152261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:30.151878 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:31.151652 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:31.151619 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:31.151833 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:31.151682 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:31.968241 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:37:31.968184 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:37:32.151731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:32.151695 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:32.151945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:32.151754 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:33.151776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:33.151702 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:33.151776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:33.151757 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:34.151692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:34.151652 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:37:34.151893 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:34.151714 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 00:37:35.520497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:35.520464 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:35.521010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:35.520825 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:35.521596 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:35.521572 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 00:37:56.123748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:37:56.123685 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:38:01.950191 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:38:01.950154 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:38:02.594930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.594895 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 00:38:02.600677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.600651 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.627811 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.627783 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-671959-serving-cert\"" Oct 31 00:38:02.628487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.628459 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:38:02.628739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.628630 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:38:02.628739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.628700 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:38:02.628919 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.628751 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\"" Oct 31 00:38:02.630260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.630241 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 00:38:02.749780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.749745 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpvx\" (UniqueName: \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.749943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.749804 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.749943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.749889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.750066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.750006 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.850605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.850505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.850783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.850644 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpvx\" (UniqueName: \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.850783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.850699 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.850783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.850729 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.851450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.851418 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.853374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.853330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.853495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.853413 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.872795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.872762 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpvx\" (UniqueName: \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\") pod \"maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:02.914583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:02.914559 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:03.218847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:03.218820 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 00:38:03.219822 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:03.219793 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74627e15_7c07_4e25_bca2_6edd558d685e.slice/crio-a7fbdc63561f88b6b00ebc74c3ae901d0cda71430645361906145c37afb53045 WatchSource:0}: Error finding container a7fbdc63561f88b6b00ebc74c3ae901d0cda71430645361906145c37afb53045: Status 404 returned error can't find the container with id a7fbdc63561f88b6b00ebc74c3ae901d0cda71430645361906145c37afb53045 Oct 31 00:38:03.419173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:03.419138 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" event={"ID":"74627e15-7c07-4e25-bca2-6edd558d685e","Type":"ContainerStarted","Data":"a7fbdc63561f88b6b00ebc74c3ae901d0cda71430645361906145c37afb53045"} Oct 31 00:38:06.438464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:06.438418 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" event={"ID":"74627e15-7c07-4e25-bca2-6edd558d685e","Type":"ContainerStarted","Data":"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a"} Oct 31 00:38:06.438865 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:06.438503 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:06.481201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:06.481153 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" podStartSLOduration=2.161297757 podStartE2EDuration="4.481140883s" totalImagesPullingTime="2.319843126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:06.479470708 +0000 UTC m=+1154.928558346" watchObservedRunningTime="2026-10-31 00:38:06.481140883 +0000 UTC m=+1154.930228521" Oct 31 00:38:08.029818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:08.029783 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:38:12.448851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:12.448821 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 00:38:17.672366 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.672331 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 00:38:17.677822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.677792 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.716808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.716780 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"default-dockercfg-cbmxc\"" Oct 31 00:38:17.717001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.716973 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"simulator-w1-b024b5-kserve-self-signed-certs\"" Oct 31 00:38:17.717173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.717153 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"openshift-service-ca.crt\"" Oct 31 00:38:17.717410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.717393 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"kube-root-ca.crt\"" Oct 31 00:38:17.720065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.720029 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 00:38:17.788712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.788682 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkq46\" (UniqueName: \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.788839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.788784 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.788899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.788872 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.788976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.788956 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.789042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.788997 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.789042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.789037 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890068 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890127 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890152 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890220 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkq46\" (UniqueName: \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890283 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890592 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890635 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.890684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.890676 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.892615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.892593 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.893023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.893002 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.913496 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.913473 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkq46\" (UniqueName: \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\") pod \"simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:17.992940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:17.992854 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:18.312427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:18.312398 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 00:38:18.313493 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:18.313466 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ef2907_6a05_4459_a039_ef9f1688b8e9.slice/crio-676a037d92326ee54a018d7e109e0aa36097eb6036be51be36cca9b4d7dfd84b WatchSource:0}: Error finding container 676a037d92326ee54a018d7e109e0aa36097eb6036be51be36cca9b4d7dfd84b: Status 404 returned error can't find the container with id 676a037d92326ee54a018d7e109e0aa36097eb6036be51be36cca9b4d7dfd84b Oct 31 00:38:18.505254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:18.505191 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" event={"ID":"91ef2907-6a05-4459-a039-ef9f1688b8e9","Type":"ContainerStarted","Data":"676a037d92326ee54a018d7e109e0aa36097eb6036be51be36cca9b4d7dfd84b"} Oct 31 00:38:19.513181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:19.513144 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" event={"ID":"91ef2907-6a05-4459-a039-ef9f1688b8e9","Type":"ContainerStarted","Data":"d22c4a3c33e38b922998418ffa113d04f1322a0921fc4d9a528245b31693bf4d"} Oct 31 00:38:19.554777 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:19.554717 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podStartSLOduration=1.447043496 podStartE2EDuration="2.554698424s" totalImagesPullingTime="1.107654928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:19.551900108 +0000 UTC m=+1168.000987748" watchObservedRunningTime="2026-10-31 00:38:19.554698424 +0000 UTC m=+1168.003786064" Oct 31 00:38:19.993452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:19.993417 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:21.016552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:21.016521 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:38:27.742593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.742561 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 00:38:27.748950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.748929 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774305 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774361 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ck8l\" (UniqueName: \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774474 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774605 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.774769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.774751 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.775942 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.775918 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"openshift-service-ca.crt\"" Oct 31 00:38:27.776114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.775921 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"kube-root-ca.crt\"" Oct 31 00:38:27.776114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.775964 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"default-dockercfg-fhc67\"" Oct 31 00:38:27.776401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.776384 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"simulator-w3-daaf26-kserve-self-signed-certs\"" Oct 31 00:38:27.778161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.778141 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 00:38:27.875782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.875749 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.875782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.875789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.875943 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876053 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876097 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ck8l\" (UniqueName: \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876128 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876238 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876330 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876268 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.876552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.876524 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.878622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.878579 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.878706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.878624 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:27.905093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.905064 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ck8l\" (UniqueName: \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\") pod \"simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:28.001284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:27.993906 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:28.010282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.009997 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:28.061474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.061437 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:28.367049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.365696 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 00:38:28.569299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.569254 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" event={"ID":"6173beed-90e9-49b3-82a8-6404a4d943c6","Type":"ContainerStarted","Data":"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f"} Oct 31 00:38:28.569482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.569308 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" event={"ID":"6173beed-90e9-49b3-82a8-6404a4d943c6","Type":"ContainerStarted","Data":"5f7280428aa8a38bb0aa67fa63554c06422dc0cd27a209a04700c6b7a8199c5f"} Oct 31 00:38:28.580803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.580779 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:38:28.621496 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:28.620321 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podStartSLOduration=1.620301846 podStartE2EDuration="1.620301846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:28.618808206 +0000 UTC m=+1177.067895845" watchObservedRunningTime="2026-10-31 00:38:28.620301846 +0000 UTC m=+1177.069389487" Oct 31 00:38:29.062511 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.062419 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:29.701698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.701667 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 00:38:29.705772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.705749 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.720686 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.720646 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"kube-root-ca.crt\"" Oct 31 00:38:29.723240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.723187 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"openshift-service-ca.crt\"" Oct 31 00:38:29.740743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.740720 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 00:38:29.748383 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.748352 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"default-dockercfg-t4mw9\"" Oct 31 00:38:29.748512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.748424 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"simulator-w2-735123-kserve-self-signed-certs\"" Oct 31 00:38:29.796288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796253 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.796427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796301 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.796477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796437 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.796518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796470 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvsq\" (UniqueName: \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.796558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796540 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.796596 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.796581 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897110 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897177 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897201 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897284 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897316 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvsq\" (UniqueName: \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897360 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897636 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897598 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.897909 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897880 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.898028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.897923 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.899566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.899540 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.900465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.900436 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:29.919676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:29.919649 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvsq\" (UniqueName: \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\") pod \"simulator-w2-735123-kserve-67b489cd8-6577f\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:30.024429 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.024339 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:30.332916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.332888 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 00:38:30.334515 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:30.334491 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a663f51_7fe6_4064_a452_4d15d4e76b3c.slice/crio-b8ee921ba9ef09e9fa4a848a32b395cc756a304fcc93fe3f149acd3c5409f5d1 WatchSource:0}: Error finding container b8ee921ba9ef09e9fa4a848a32b395cc756a304fcc93fe3f149acd3c5409f5d1: Status 404 returned error can't find the container with id b8ee921ba9ef09e9fa4a848a32b395cc756a304fcc93fe3f149acd3c5409f5d1 Oct 31 00:38:30.430620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.430515 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:38:30.583497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.583412 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" event={"ID":"5a663f51-7fe6-4064-a452-4d15d4e76b3c","Type":"ContainerStarted","Data":"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438"} Oct 31 00:38:30.583497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.583458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" event={"ID":"5a663f51-7fe6-4064-a452-4d15d4e76b3c","Type":"ContainerStarted","Data":"b8ee921ba9ef09e9fa4a848a32b395cc756a304fcc93fe3f149acd3c5409f5d1"} Oct 31 00:38:30.636729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:30.636677 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podStartSLOduration=1.636662276 podStartE2EDuration="1.636662276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:30.635595299 +0000 UTC m=+1179.084682938" watchObservedRunningTime="2026-10-31 00:38:30.636662276 +0000 UTC m=+1179.085749918" Oct 31 00:38:31.025226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:31.025124 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:31.948907 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:38:31.948867 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:38:35.452088 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.452038 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 00:38:35.460021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.459994 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.474159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.474121 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 00:38:35.483242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.481523 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"simulator-w5-c9e377-kserve-self-signed-certs\"" Oct 31 00:38:35.483242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.481681 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"default-dockercfg-m7xjg\"" Oct 31 00:38:35.483688 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.483661 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"openshift-service-ca.crt\"" Oct 31 00:38:35.483823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.483731 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9e377\"/\"kube-root-ca.crt\"" Oct 31 00:38:35.553237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553195 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.553411 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.553411 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553395 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.553536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.553536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553449 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.553643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.553584 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vpl\" (UniqueName: \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.654982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.654943 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655105 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655202 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655250 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655280 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655319 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vpl\" (UniqueName: \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655431 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655631 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.655840 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.655821 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.657550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.657524 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.657794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.657778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.677315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.677281 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vpl\" (UniqueName: \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\") pod \"simulator-w5-c9e377-kserve-86bb57c78-tdpkn\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:35.775757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:35.775670 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:36.081362 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:36.081335 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69adc0a6_30aa_4c5a_a120_656f7d569ca8.slice/crio-b813f640797c4a5116fac0d7bd7e7ffd4031bb988e2a21a514f3ca54672ecf27 WatchSource:0}: Error finding container b813f640797c4a5116fac0d7bd7e7ffd4031bb988e2a21a514f3ca54672ecf27: Status 404 returned error can't find the container with id b813f640797c4a5116fac0d7bd7e7ffd4031bb988e2a21a514f3ca54672ecf27 Oct 31 00:38:36.083944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:36.083917 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 00:38:36.620085 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:36.620046 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" event={"ID":"69adc0a6-30aa-4c5a-a120-656f7d569ca8","Type":"ContainerStarted","Data":"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971"} Oct 31 00:38:36.620480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:36.620099 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" event={"ID":"69adc0a6-30aa-4c5a-a120-656f7d569ca8","Type":"ContainerStarted","Data":"b813f640797c4a5116fac0d7bd7e7ffd4031bb988e2a21a514f3ca54672ecf27"} Oct 31 00:38:36.668767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:36.668709 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podStartSLOduration=1.6686898989999999 podStartE2EDuration="1.668689899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:36.665627347 +0000 UTC m=+1185.114714986" watchObservedRunningTime="2026-10-31 00:38:36.668689899 +0000 UTC m=+1185.117777540" Oct 31 00:38:36.776142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:36.776107 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:38.062434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.062393 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:38.075399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.075368 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:38.643247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.643217 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:38:38.688371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.688332 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 00:38:38.693371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.693347 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.710657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.710623 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"default-dockercfg-bqtkx\"" Oct 31 00:38:38.710809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.710689 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"openshift-service-ca.crt\"" Oct 31 00:38:38.710809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.710751 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"kube-root-ca.crt\"" Oct 31 00:38:38.710925 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.710849 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"simulator-w6-be301b-kserve-self-signed-certs\"" Oct 31 00:38:38.712373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.712345 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 00:38:38.787764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.787710 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mld52\" (UniqueName: \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.788044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.788020 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.788143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.788076 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.788143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.788101 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.788267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.788241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.788335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.788318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889380 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889446 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889473 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889633 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889777 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889726 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889805 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mld52\" (UniqueName: \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889863 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.889894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.889807 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.890060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.890039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.892136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.892114 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.892307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.892284 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:38.911863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:38.911807 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mld52\" (UniqueName: \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\") pod \"simulator-w6-be301b-kserve-74c7795fc8-zdshx\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:39.015134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:39.015094 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:39.322153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:39.322128 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 00:38:39.323695 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:39.323670 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a96945f_f289_4cd4_ad20_d21439cb75e9.slice/crio-9a501c87adea97932ad64d048dce7d2e002025c8bf85a217289f1ce57df31b90 WatchSource:0}: Error finding container 9a501c87adea97932ad64d048dce7d2e002025c8bf85a217289f1ce57df31b90: Status 404 returned error can't find the container with id 9a501c87adea97932ad64d048dce7d2e002025c8bf85a217289f1ce57df31b90 Oct 31 00:38:39.641933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:39.641832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" event={"ID":"8a96945f-f289-4cd4-ad20-d21439cb75e9","Type":"ContainerStarted","Data":"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef"} Oct 31 00:38:39.641933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:39.641885 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" event={"ID":"8a96945f-f289-4cd4-ad20-d21439cb75e9","Type":"ContainerStarted","Data":"9a501c87adea97932ad64d048dce7d2e002025c8bf85a217289f1ce57df31b90"} Oct 31 00:38:39.687185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:39.687122 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podStartSLOduration=1.6871044899999998 podStartE2EDuration="1.68710449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:39.683910599 +0000 UTC m=+1188.132998253" watchObservedRunningTime="2026-10-31 00:38:39.68710449 +0000 UTC m=+1188.136192130" Oct 31 00:38:40.015911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:40.015827 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:40.024460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:40.024423 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:40.039944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:40.039918 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:40.661400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:40.661367 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:38:45.048030 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:45.047997 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:38:45.776675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:45.776634 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:45.790853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:45.790825 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:46.697594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:46.697565 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:38:48.292041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.292009 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 00:38:48.295892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.295870 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.321615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.321586 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"simulator-w4-0d8bae-kserve-self-signed-certs\"" Oct 31 00:38:48.321929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.321909 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"openshift-service-ca.crt\"" Oct 31 00:38:48.322040 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.321972 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"default-dockercfg-422tf\"" Oct 31 00:38:48.324923 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.324898 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 00:38:48.325127 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.325111 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"kube-root-ca.crt\"" Oct 31 00:38:48.392782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.392745 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.392947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.392792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.392947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.392813 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.393063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.392952 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxj75\" (UniqueName: \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.393063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.392984 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.393063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.393015 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494263 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494325 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494348 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494409 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxj75\" (UniqueName: \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494441 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494484 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.494841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494812 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.495017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.494833 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.495124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.495027 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.497255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.497229 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.497852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.497828 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.519605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.519570 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxj75\" (UniqueName: \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\") pod \"simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.610556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.610523 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:48.914499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:48.914472 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 00:38:48.915781 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:48.915755 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d447db4_0d26_47da_b86c_efdaa1bc173c.slice/crio-eebc444b34bddb912a4c727f3952409c9c9986970cbf70e0ab5f52fae6284960 WatchSource:0}: Error finding container eebc444b34bddb912a4c727f3952409c9c9986970cbf70e0ab5f52fae6284960: Status 404 returned error can't find the container with id eebc444b34bddb912a4c727f3952409c9c9986970cbf70e0ab5f52fae6284960 Oct 31 00:38:49.016283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.016255 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:49.028928 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.028905 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:49.719044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.719007 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" event={"ID":"1d447db4-0d26-47da-b86c-efdaa1bc173c","Type":"ContainerStarted","Data":"b329fd9d0c466b79230d3d133524d854d78d974287b0f4f5766b6f97484dcfbe"} Oct 31 00:38:49.719575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.719057 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" event={"ID":"1d447db4-0d26-47da-b86c-efdaa1bc173c","Type":"ContainerStarted","Data":"eebc444b34bddb912a4c727f3952409c9c9986970cbf70e0ab5f52fae6284960"} Oct 31 00:38:49.735324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.735294 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:38:49.757063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:49.757007 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podStartSLOduration=1.75699229 podStartE2EDuration="1.75699229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:49.756615129 +0000 UTC m=+1198.205702765" watchObservedRunningTime="2026-10-31 00:38:49.75699229 +0000 UTC m=+1198.206079930" Oct 31 00:38:50.611499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:50.611470 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:51.842064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:51.842038 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:38:51.845433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:51.845412 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:38:56.149092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.149055 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 00:38:56.153112 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.153092 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.169864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.169841 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 00:38:56.177432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.177408 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"premium-w1-b024b5-kserve-self-signed-certs\"" Oct 31 00:38:56.181810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.181782 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.181926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.181831 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.181991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.181921 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.181991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.181946 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.182099 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.182055 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5rd\" (UniqueName: \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.182161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.182097 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.283390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283344 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5rd\" (UniqueName: \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.283390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283397 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.283643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283459 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.283643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283604 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.283938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283914 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.284006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.283923 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.284235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.284183 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.284323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.284252 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.284523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.284489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.286460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.286423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.286888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.286867 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.305926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.305895 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5rd\" (UniqueName: \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\") pod \"premium-w1-b024b5-kserve-5bc74d857d-8gxhf\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.468084 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.467996 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:56.767620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:56.767593 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 00:38:56.768307 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:38:56.768277 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c04352b_c078_45e6_bc97_110fb00e5d37.slice/crio-4a31ad9c34e581591f066bb9440334b0636a9b3c899e5bf70d270149588876e7 WatchSource:0}: Error finding container 4a31ad9c34e581591f066bb9440334b0636a9b3c899e5bf70d270149588876e7: Status 404 returned error can't find the container with id 4a31ad9c34e581591f066bb9440334b0636a9b3c899e5bf70d270149588876e7 Oct 31 00:38:57.775518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:57.775483 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" event={"ID":"1c04352b-c078-45e6-bc97-110fb00e5d37","Type":"ContainerStarted","Data":"599fbcc77c9679bfc9a9fa0d0850ab0413f6f455bd8209e5ab097f5d66cbcf4f"} Oct 31 00:38:57.775887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:57.775524 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" event={"ID":"1c04352b-c078-45e6-bc97-110fb00e5d37","Type":"ContainerStarted","Data":"4a31ad9c34e581591f066bb9440334b0636a9b3c899e5bf70d270149588876e7"} Oct 31 00:38:57.814734 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:57.814681 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podStartSLOduration=1.814668507 podStartE2EDuration="1.814668507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:57.814078389 +0000 UTC m=+1206.263166031" watchObservedRunningTime="2026-10-31 00:38:57.814668507 +0000 UTC m=+1206.263756146" Oct 31 00:38:58.468303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:58.468270 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:38:58.611683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:58.611641 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:58.625397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:58.625370 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:38:58.792263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:38:58.792172 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:39:00.623969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:00.623936 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:39:01.949799 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:39:01.949761 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:39:06.469311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:06.469138 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:39:06.482135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:06.482110 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:39:06.844697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:06.844665 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:39:12.357283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.357247 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 00:39:12.360943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.360923 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.385958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.385921 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"premium-w2-735123-kserve-self-signed-certs\"" Oct 31 00:39:12.389821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.389790 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 00:39:12.456560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456526 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.456786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456638 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.456786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456697 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7f67\" (UniqueName: \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.456786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456736 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.456958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456860 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.456958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.456882 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.562795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.562751 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563166 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563321 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563387 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7f67\" (UniqueName: \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563451 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563528 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563558 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.563910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.563889 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.578305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.578277 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.578791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.578749 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.595237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.591449 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7f67\" (UniqueName: \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\") pod \"premium-w2-735123-kserve-6fdd4bfc5c-rc7fs\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.678022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.677936 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:12.993895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:12.993864 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 00:39:12.994873 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:39:12.994843 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod447a3da0_79c8_482d_9da8_e64356f114df.slice/crio-f3e9df2dffcf896bdb1c9d6addc5ce41d7894385606a888f77bfe0b85a7e12b0 WatchSource:0}: Error finding container f3e9df2dffcf896bdb1c9d6addc5ce41d7894385606a888f77bfe0b85a7e12b0: Status 404 returned error can't find the container with id f3e9df2dffcf896bdb1c9d6addc5ce41d7894385606a888f77bfe0b85a7e12b0 Oct 31 00:39:13.878231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:13.878169 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" event={"ID":"447a3da0-79c8-482d-9da8-e64356f114df","Type":"ContainerStarted","Data":"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0"} Oct 31 00:39:13.878776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:13.878242 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" event={"ID":"447a3da0-79c8-482d-9da8-e64356f114df","Type":"ContainerStarted","Data":"f3e9df2dffcf896bdb1c9d6addc5ce41d7894385606a888f77bfe0b85a7e12b0"} Oct 31 00:39:13.918475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:13.918379 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podStartSLOduration=1.9183662639999999 podStartE2EDuration="1.918366264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:13.916932484 +0000 UTC m=+1222.366020124" watchObservedRunningTime="2026-10-31 00:39:13.918366264 +0000 UTC m=+1222.367453903" Oct 31 00:39:14.626648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:14.626606 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:39:14.678163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:14.678128 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:22.678541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:22.678508 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:22.691593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:22.691566 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:22.939473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:22.939391 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:39:27.925422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:27.925386 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:39:31.947328 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:39:31.947287 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:39:47.589483 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.589447 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf"] Oct 31 00:39:47.593381 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.593364 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.611797 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.611774 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf"] Oct 31 00:39:47.619382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.619360 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:47.619535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.619397 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:39:47.725756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.725724 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.725945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.725825 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.725945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.725890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhlx\" (UniqueName: \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.826420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.826382 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.826610 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.826492 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.826610 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.826561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhlx\" (UniqueName: \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.827309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.827280 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.828937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.828900 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.848497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.848426 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhlx\" (UniqueName: \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\") pod \"maas-api-revoke-sub-42a7b93c-dzbwf\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.907018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.906986 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:47.908761 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:47.908735 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:39:48.209578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:48.209548 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf"] Oct 31 00:39:48.209948 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:39:48.209915 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b403f37_1e3f_4bd1_a369_56d7ec847455.slice/crio-c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52 WatchSource:0}: Error finding container c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52: Status 404 returned error can't find the container with id c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52 Oct 31 00:39:48.213733 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:48.213711 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:39:49.091090 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:49.091052 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" event={"ID":"3b403f37-1e3f-4bd1-a369-56d7ec847455","Type":"ContainerStarted","Data":"c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52"} Oct 31 00:39:51.105020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:51.104981 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" event={"ID":"3b403f37-1e3f-4bd1-a369-56d7ec847455","Type":"ContainerStarted","Data":"cd6d677f6523f54a82c67229b4f7437c7cd1315a6db88ba44d30ed819d36b3d5"} Oct 31 00:39:51.197915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:51.197828 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" podStartSLOduration=1.764196139 podStartE2EDuration="4.197807656s" totalImagesPullingTime="2.433611517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:51.195019186 +0000 UTC m=+1259.644106824" watchObservedRunningTime="2026-10-31 00:39:51.197807656 +0000 UTC m=+1259.646895294" Oct 31 00:39:52.111633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:52.111601 5011 generic.go:356] "Generic (PLEG): container finished" podID="3b403f37-1e3f-4bd1-a369-56d7ec847455" containerID="cd6d677f6523f54a82c67229b4f7437c7cd1315a6db88ba44d30ed819d36b3d5" exitCode=0 Oct 31 00:39:52.112011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:52.111684 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" event={"ID":"3b403f37-1e3f-4bd1-a369-56d7ec847455","Type":"ContainerDied","Data":"cd6d677f6523f54a82c67229b4f7437c7cd1315a6db88ba44d30ed819d36b3d5"} Oct 31 00:39:53.255944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.255919 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:53.382115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.382025 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\") pod \"3b403f37-1e3f-4bd1-a369-56d7ec847455\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " Oct 31 00:39:53.382307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.382179 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\" (UniqueName: \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\") pod \"3b403f37-1e3f-4bd1-a369-56d7ec847455\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " Oct 31 00:39:53.382307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.382233 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\") pod \"3b403f37-1e3f-4bd1-a369-56d7ec847455\" (UID: \"3b403f37-1e3f-4bd1-a369-56d7ec847455\") " Oct 31 00:39:53.382829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.382797 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca" pod "3b403f37-1e3f-4bd1-a369-56d7ec847455" (UID: "3b403f37-1e3f-4bd1-a369-56d7ec847455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:53.384591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.384564 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca" pod "3b403f37-1e3f-4bd1-a369-56d7ec847455" (UID: "3b403f37-1e3f-4bd1-a369-56d7ec847455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:53.384702 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.384594 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx" pod "3b403f37-1e3f-4bd1-a369-56d7ec847455" (UID: "3b403f37-1e3f-4bd1-a369-56d7ec847455"). InnerVolumeSpecName "kube-api-access-dlhlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:53.484042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.484007 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:39:53.484042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.484044 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhlx\" (UniqueName: \"kubernetes.io/projected/3b403f37-1e3f-4bd1-a369-56d7ec847455-kube-api-access-dlhlx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:39:53.484262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:53.484059 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b403f37-1e3f-4bd1-a369-56d7ec847455-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:39:54.125954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:54.125922 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" Oct 31 00:39:54.126128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:54.125955 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf" event={"ID":"3b403f37-1e3f-4bd1-a369-56d7ec847455","Type":"ContainerDied","Data":"c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52"} Oct 31 00:39:54.126128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:39:54.126009 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0d47c081fdd01f069b0691e9576cc2ca1e5b8a9c2dc2823acb74edf77d47b52" Oct 31 00:40:01.947066 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:40:01.947031 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:40:05.891247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.891198 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k"] Oct 31 00:40:05.891700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.891686 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3b403f37-1e3f-4bd1-a369-56d7ec847455" containerName="revoke-keys" Oct 31 00:40:05.891753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.891702 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b403f37-1e3f-4bd1-a369-56d7ec847455" containerName="revoke-keys" Oct 31 00:40:05.891799 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.891789 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b403f37-1e3f-4bd1-a369-56d7ec847455" containerName="revoke-keys" Oct 31 00:40:05.895276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.895259 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:05.914785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.914758 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k"] Oct 31 00:40:05.922138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.922094 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:05.922292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:05.922252 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:06.004323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.004295 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.004466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.004396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.004507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.004466 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p92g\" (UniqueName: \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.106446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.106408 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.106608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.106501 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.106608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.106567 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p92g\" (UniqueName: \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.108266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.108237 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.115456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.115428 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.128868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.128837 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p92g\" (UniqueName: \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\") pod \"maas-api-revoke-sub-2798879a-kg22k\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.207419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.207333 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:06.514343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:06.514316 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k"] Oct 31 00:40:06.515832 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:06.515806 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe13aa_dfb5_4141_83c6_9575aff00a74.slice/crio-0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f WatchSource:0}: Error finding container 0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f: Status 404 returned error can't find the container with id 0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f Oct 31 00:40:07.208240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:07.208146 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" event={"ID":"fefe13aa-dfb5-4141-83c6-9575aff00a74","Type":"ContainerStarted","Data":"e3ba72afc6af3c8919477e07b914a8ad4b65f36a2f50d77e143c4e1cb3430277"} Oct 31 00:40:07.208240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:07.208196 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" event={"ID":"fefe13aa-dfb5-4141-83c6-9575aff00a74","Type":"ContainerStarted","Data":"0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f"} Oct 31 00:40:07.250864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:07.250817 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" podStartSLOduration=2.250803332 podStartE2EDuration="2.250803332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:07.247991831 +0000 UTC m=+1275.697079471" watchObservedRunningTime="2026-10-31 00:40:07.250803332 +0000 UTC m=+1275.699890970" Oct 31 00:40:08.215294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:08.215261 5011 generic.go:356] "Generic (PLEG): container finished" podID="fefe13aa-dfb5-4141-83c6-9575aff00a74" containerID="e3ba72afc6af3c8919477e07b914a8ad4b65f36a2f50d77e143c4e1cb3430277" exitCode=0 Oct 31 00:40:08.215687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:08.215347 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" event={"ID":"fefe13aa-dfb5-4141-83c6-9575aff00a74","Type":"ContainerDied","Data":"e3ba72afc6af3c8919477e07b914a8ad4b65f36a2f50d77e143c4e1cb3430277"} Oct 31 00:40:09.382007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.381981 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:09.438813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.438782 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\") pod \"fefe13aa-dfb5-4141-83c6-9575aff00a74\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " Oct 31 00:40:09.438957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.438824 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\" (UniqueName: \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\") pod \"fefe13aa-dfb5-4141-83c6-9575aff00a74\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " Oct 31 00:40:09.438957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.438857 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\") pod \"fefe13aa-dfb5-4141-83c6-9575aff00a74\" (UID: \"fefe13aa-dfb5-4141-83c6-9575aff00a74\") " Oct 31 00:40:09.439232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.439189 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca" pod "fefe13aa-dfb5-4141-83c6-9575aff00a74" (UID: "fefe13aa-dfb5-4141-83c6-9575aff00a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:09.440904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.440873 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca" pod "fefe13aa-dfb5-4141-83c6-9575aff00a74" (UID: "fefe13aa-dfb5-4141-83c6-9575aff00a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:09.441036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.441019 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g" pod "fefe13aa-dfb5-4141-83c6-9575aff00a74" (UID: "fefe13aa-dfb5-4141-83c6-9575aff00a74"). InnerVolumeSpecName "kube-api-access-7p92g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:09.539708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.539636 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:09.539708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.539664 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p92g\" (UniqueName: \"kubernetes.io/projected/fefe13aa-dfb5-4141-83c6-9575aff00a74-kube-api-access-7p92g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:09.539708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:09.539675 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefe13aa-dfb5-4141-83c6-9575aff00a74-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:10.232339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:10.232294 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" event={"ID":"fefe13aa-dfb5-4141-83c6-9575aff00a74","Type":"ContainerDied","Data":"0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f"} Oct 31 00:40:10.232512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:10.232394 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e066128f7b1695c6f0564782ac9bd88798531c2f5e33a9899a01f86a7d57a5f" Oct 31 00:40:10.232512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:10.232309 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k" Oct 31 00:40:16.084325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:16.084289 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:40:21.857492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.857456 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s"] Oct 31 00:40:21.863293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.857959 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fefe13aa-dfb5-4141-83c6-9575aff00a74" containerName="revoke-keys" Oct 31 00:40:21.863293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.857973 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe13aa-dfb5-4141-83c6-9575aff00a74" containerName="revoke-keys" Oct 31 00:40:21.863293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.858092 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe13aa-dfb5-4141-83c6-9575aff00a74" containerName="revoke-keys" Oct 31 00:40:21.864066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.864050 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:21.884189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.884158 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s"] Oct 31 00:40:21.890587 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.890562 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:21.890694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.890631 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:21.959507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.959473 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:21.959704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.959526 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:21.959704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:21.959647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxh9\" (UniqueName: \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.060298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.060264 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.060492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.060328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxh9\" (UniqueName: \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.060492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.060417 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.060996 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.060974 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.062739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.062721 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.082227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.082188 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxh9\" (UniqueName: \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\") pod \"maas-api-revoke-sub-89025758-rxc7s\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.177450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.177371 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:22.480929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:22.480898 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s"] Oct 31 00:40:22.481076 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:22.480940 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62b99a5_19e7_41f4_912d_f888f9773aee.slice/crio-c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb WatchSource:0}: Error finding container c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb: Status 404 returned error can't find the container with id c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb Oct 31 00:40:23.311967 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.311922 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" event={"ID":"d62b99a5-19e7-41f4-912d-f888f9773aee","Type":"ContainerStarted","Data":"a843fbca1a9e9f392383e4bc169d25f36c07667e366da88b47b1b7a08dce548e"} Oct 31 00:40:23.312403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.311992 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" event={"ID":"d62b99a5-19e7-41f4-912d-f888f9773aee","Type":"ContainerStarted","Data":"c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb"} Oct 31 00:40:23.350132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.350085 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" podStartSLOduration=2.35007055 podStartE2EDuration="2.35007055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:23.349074019 +0000 UTC m=+1291.798161658" watchObservedRunningTime="2026-10-31 00:40:23.35007055 +0000 UTC m=+1291.799158211" Oct 31 00:40:23.804917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.804881 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg"] Oct 31 00:40:23.808750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.808732 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.843104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.843069 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg"] Oct 31 00:40:23.885852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.885819 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.886006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.885858 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92mtt\" (UniqueName: \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.886006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.885878 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.986687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.986653 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.986837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.986692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92mtt\" (UniqueName: \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.986837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.986714 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.987515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.987488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:23.988972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:23.988951 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:24.019781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:24.019751 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92mtt\" (UniqueName: \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\") pod \"maas-api-revoke-sub-42319158-zhqpg\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:24.122016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:24.121993 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:24.318798 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:24.318759 5011 generic.go:356] "Generic (PLEG): container finished" podID="d62b99a5-19e7-41f4-912d-f888f9773aee" containerID="a843fbca1a9e9f392383e4bc169d25f36c07667e366da88b47b1b7a08dce548e" exitCode=0 Oct 31 00:40:24.319301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:24.318856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" event={"ID":"d62b99a5-19e7-41f4-912d-f888f9773aee","Type":"ContainerDied","Data":"a843fbca1a9e9f392383e4bc169d25f36c07667e366da88b47b1b7a08dce548e"} Oct 31 00:40:24.426258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:24.426225 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg"] Oct 31 00:40:24.427408 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:24.427375 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116b894a_8e1b_4ca6_a026_43eb9b62ea77.slice/crio-7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1 WatchSource:0}: Error finding container 7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1: Status 404 returned error can't find the container with id 7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1 Oct 31 00:40:25.327194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.327163 5011 generic.go:356] "Generic (PLEG): container finished" podID="116b894a-8e1b-4ca6-a026-43eb9b62ea77" containerID="2047ac0c73c233acbdd4e97d71718a84b56d06329a4837bc7e486569a6b53609" exitCode=0 Oct 31 00:40:25.327605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.327266 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" event={"ID":"116b894a-8e1b-4ca6-a026-43eb9b62ea77","Type":"ContainerDied","Data":"2047ac0c73c233acbdd4e97d71718a84b56d06329a4837bc7e486569a6b53609"} Oct 31 00:40:25.327605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.327422 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" event={"ID":"116b894a-8e1b-4ca6-a026-43eb9b62ea77","Type":"ContainerStarted","Data":"7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1"} Oct 31 00:40:25.487979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.487952 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:25.605288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.605255 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\") pod \"d62b99a5-19e7-41f4-912d-f888f9773aee\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " Oct 31 00:40:25.605440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.605299 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\") pod \"d62b99a5-19e7-41f4-912d-f888f9773aee\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " Oct 31 00:40:25.605440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.605401 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\" (UniqueName: \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\") pod \"d62b99a5-19e7-41f4-912d-f888f9773aee\" (UID: \"d62b99a5-19e7-41f4-912d-f888f9773aee\") " Oct 31 00:40:25.605579 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.605552 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca" pod "d62b99a5-19e7-41f4-912d-f888f9773aee" (UID: "d62b99a5-19e7-41f4-912d-f888f9773aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:25.607304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.607278 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca" pod "d62b99a5-19e7-41f4-912d-f888f9773aee" (UID: "d62b99a5-19e7-41f4-912d-f888f9773aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:25.607410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.607391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9" pod "d62b99a5-19e7-41f4-912d-f888f9773aee" (UID: "d62b99a5-19e7-41f4-912d-f888f9773aee"). InnerVolumeSpecName "kube-api-access-4nxh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:25.706298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.706268 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:25.706298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.706294 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62b99a5-19e7-41f4-912d-f888f9773aee-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:25.706298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:25.706303 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxh9\" (UniqueName: \"kubernetes.io/projected/d62b99a5-19e7-41f4-912d-f888f9773aee-kube-api-access-4nxh9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:26.333989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.333958 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" Oct 31 00:40:26.334395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.333987 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s" event={"ID":"d62b99a5-19e7-41f4-912d-f888f9773aee","Type":"ContainerDied","Data":"c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb"} Oct 31 00:40:26.334395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.334032 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c22c1516508fa1ec738b0ffa55581107c206c6799c049532294a43e82f7c0bfb" Oct 31 00:40:26.482284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.482257 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:26.616392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.616306 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\") pod \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " Oct 31 00:40:26.616392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.616388 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\" (UniqueName: \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\") pod \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " Oct 31 00:40:26.616630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.616490 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\") pod \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\" (UID: \"116b894a-8e1b-4ca6-a026-43eb9b62ea77\") " Oct 31 00:40:26.616877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.616846 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca" pod "116b894a-8e1b-4ca6-a026-43eb9b62ea77" (UID: "116b894a-8e1b-4ca6-a026-43eb9b62ea77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:26.618650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.618621 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt" pod "116b894a-8e1b-4ca6-a026-43eb9b62ea77" (UID: "116b894a-8e1b-4ca6-a026-43eb9b62ea77"). InnerVolumeSpecName "kube-api-access-92mtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:26.618881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.618860 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca" pod "116b894a-8e1b-4ca6-a026-43eb9b62ea77" (UID: "116b894a-8e1b-4ca6-a026-43eb9b62ea77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:26.717828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.717799 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:26.717828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.717827 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116b894a-8e1b-4ca6-a026-43eb9b62ea77-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:26.717828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:26.717836 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92mtt\" (UniqueName: \"kubernetes.io/projected/116b894a-8e1b-4ca6-a026-43eb9b62ea77-kube-api-access-92mtt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:27.093708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.093674 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m"] Oct 31 00:40:27.094113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094099 5011 cpu_manager.go:414] "containerMap: removing container" podUID="116b894a-8e1b-4ca6-a026-43eb9b62ea77" containerName="revoke-keys" Oct 31 00:40:27.094113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094113 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116b894a-8e1b-4ca6-a026-43eb9b62ea77" containerName="revoke-keys" Oct 31 00:40:27.094264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094142 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d62b99a5-19e7-41f4-912d-f888f9773aee" containerName="revoke-keys" Oct 31 00:40:27.094264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094148 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62b99a5-19e7-41f4-912d-f888f9773aee" containerName="revoke-keys" Oct 31 00:40:27.094264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094243 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="116b894a-8e1b-4ca6-a026-43eb9b62ea77" containerName="revoke-keys" Oct 31 00:40:27.094264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.094257 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62b99a5-19e7-41f4-912d-f888f9773aee" containerName="revoke-keys" Oct 31 00:40:27.097784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.097765 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.116461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.116438 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m"] Oct 31 00:40:27.222891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.222857 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfqv\" (UniqueName: \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.222891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.222893 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.223084 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.223033 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.324027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.323997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.324198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.324093 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfqv\" (UniqueName: \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.324198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.324112 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.324667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.324644 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.326460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.326440 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.342183 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.342158 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" Oct 31 00:40:27.342183 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.342165 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg" event={"ID":"116b894a-8e1b-4ca6-a026-43eb9b62ea77","Type":"ContainerDied","Data":"7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1"} Oct 31 00:40:27.342573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.342251 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a82eb1acb6e2ffcf882cde067ebadc365eee2607f233cd7e96a353d203caec1" Oct 31 00:40:27.345840 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.345791 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfqv\" (UniqueName: \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\") pod \"maas-api-revoke-sub-56ab8e7c-m8h6m\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.411525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.411501 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:27.721099 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.721055 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m"] Oct 31 00:40:27.722490 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:27.722455 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4873fe09_d4ba_43e9_b1a3_5db525b3ec69.slice/crio-aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb WatchSource:0}: Error finding container aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb: Status 404 returned error can't find the container with id aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb Oct 31 00:40:27.850946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.850917 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w"] Oct 31 00:40:27.856001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.855981 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:27.878981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.878956 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w"] Oct 31 00:40:27.930269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.930239 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmzz\" (UniqueName: \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:27.930448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.930297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:27.930448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:27.930321 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.031599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.031527 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmzz\" (UniqueName: \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.031744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.031612 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.031744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.031641 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.032266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.032242 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.033909 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.033886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.056481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.056455 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmzz\" (UniqueName: \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\") pod \"maas-api-revoke-sub-e8258fa1-kdl8w\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.169248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.169223 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:28.350882 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.350840 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" event={"ID":"4873fe09-d4ba-43e9-b1a3-5db525b3ec69","Type":"ContainerStarted","Data":"8411b536bb2fbceffe636fdc81b1bacb442ff11d8a98af60cc16136460247786"} Oct 31 00:40:28.351363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.350908 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" event={"ID":"4873fe09-d4ba-43e9-b1a3-5db525b3ec69","Type":"ContainerStarted","Data":"aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb"} Oct 31 00:40:28.390301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.390258 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" podStartSLOduration=1.390244622 podStartE2EDuration="1.390244622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:28.388442631 +0000 UTC m=+1296.837530272" watchObservedRunningTime="2026-10-31 00:40:28.390244622 +0000 UTC m=+1296.839332260" Oct 31 00:40:28.473635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:28.473609 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w"] Oct 31 00:40:28.475136 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:28.475113 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54fab01_0d15_4316_b6b8_370069bd01e9.slice/crio-356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167 WatchSource:0}: Error finding container 356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167: Status 404 returned error can't find the container with id 356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167 Oct 31 00:40:29.359957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:29.359923 5011 generic.go:356] "Generic (PLEG): container finished" podID="f54fab01-0d15-4316-b6b8-370069bd01e9" containerID="5d73501634ed93ab315b23154800b2227e3ca0ed8829a73a9ec32dd2fccca369" exitCode=0 Oct 31 00:40:29.360323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:29.360013 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" event={"ID":"f54fab01-0d15-4316-b6b8-370069bd01e9","Type":"ContainerDied","Data":"5d73501634ed93ab315b23154800b2227e3ca0ed8829a73a9ec32dd2fccca369"} Oct 31 00:40:29.360323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:29.360062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" event={"ID":"f54fab01-0d15-4316-b6b8-370069bd01e9","Type":"ContainerStarted","Data":"356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167"} Oct 31 00:40:29.362142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:29.362116 5011 generic.go:356] "Generic (PLEG): container finished" podID="4873fe09-d4ba-43e9-b1a3-5db525b3ec69" containerID="8411b536bb2fbceffe636fdc81b1bacb442ff11d8a98af60cc16136460247786" exitCode=0 Oct 31 00:40:29.362279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:29.362233 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" event={"ID":"4873fe09-d4ba-43e9-b1a3-5db525b3ec69","Type":"ContainerDied","Data":"8411b536bb2fbceffe636fdc81b1bacb442ff11d8a98af60cc16136460247786"} Oct 31 00:40:30.531367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.531339 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:30.551349 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.551319 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:30.662048 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.661968 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\" (UniqueName: \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\") pod \"f54fab01-0d15-4316-b6b8-370069bd01e9\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " Oct 31 00:40:30.662048 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662033 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\") pod \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " Oct 31 00:40:30.662273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662103 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\") pod \"f54fab01-0d15-4316-b6b8-370069bd01e9\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " Oct 31 00:40:30.662273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662135 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\") pod \"f54fab01-0d15-4316-b6b8-370069bd01e9\" (UID: \"f54fab01-0d15-4316-b6b8-370069bd01e9\") " Oct 31 00:40:30.662273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662170 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\" (UniqueName: \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\") pod \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " Oct 31 00:40:30.662436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662307 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\") pod \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\" (UID: \"4873fe09-d4ba-43e9-b1a3-5db525b3ec69\") " Oct 31 00:40:30.662531 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662504 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca" pod "4873fe09-d4ba-43e9-b1a3-5db525b3ec69" (UID: "4873fe09-d4ba-43e9-b1a3-5db525b3ec69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:30.662598 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662575 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca" pod "f54fab01-0d15-4316-b6b8-370069bd01e9" (UID: "f54fab01-0d15-4316-b6b8-370069bd01e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:30.662714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662698 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.662783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.662716 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.664717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.664669 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca" pod "4873fe09-d4ba-43e9-b1a3-5db525b3ec69" (UID: "4873fe09-d4ba-43e9-b1a3-5db525b3ec69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:30.665117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.665083 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv" pod "4873fe09-d4ba-43e9-b1a3-5db525b3ec69" (UID: "4873fe09-d4ba-43e9-b1a3-5db525b3ec69"). InnerVolumeSpecName "kube-api-access-sxfqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:30.665522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.665155 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca" pod "f54fab01-0d15-4316-b6b8-370069bd01e9" (UID: "f54fab01-0d15-4316-b6b8-370069bd01e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:30.665522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.665337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz" pod "f54fab01-0d15-4316-b6b8-370069bd01e9" (UID: "f54fab01-0d15-4316-b6b8-370069bd01e9"). InnerVolumeSpecName "kube-api-access-9jmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:30.693903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.693870 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f"] Oct 31 00:40:30.694473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694449 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f54fab01-0d15-4316-b6b8-370069bd01e9" containerName="revoke-keys" Oct 31 00:40:30.694473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694468 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54fab01-0d15-4316-b6b8-370069bd01e9" containerName="revoke-keys" Oct 31 00:40:30.694637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694508 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4873fe09-d4ba-43e9-b1a3-5db525b3ec69" containerName="revoke-keys" Oct 31 00:40:30.694637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694514 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4873fe09-d4ba-43e9-b1a3-5db525b3ec69" containerName="revoke-keys" Oct 31 00:40:30.694637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694591 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4873fe09-d4ba-43e9-b1a3-5db525b3ec69" containerName="revoke-keys" Oct 31 00:40:30.694637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.694604 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54fab01-0d15-4316-b6b8-370069bd01e9" containerName="revoke-keys" Oct 31 00:40:30.698125 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.698106 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.715953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.715931 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f"] Oct 31 00:40:30.764109 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764078 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.764280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764118 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgw6r\" (UniqueName: \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.764280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764235 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.764397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764345 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.764397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764358 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmzz\" (UniqueName: \"kubernetes.io/projected/f54fab01-0d15-4316-b6b8-370069bd01e9-kube-api-access-9jmzz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.764397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764368 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fab01-0d15-4316-b6b8-370069bd01e9-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.764397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.764379 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfqv\" (UniqueName: \"kubernetes.io/projected/4873fe09-d4ba-43e9-b1a3-5db525b3ec69-kube-api-access-sxfqv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.865374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.865343 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.865374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.865378 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgw6r\" (UniqueName: \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.865558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.865501 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.865965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.865948 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.867895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.867868 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:30.887024 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:30.887004 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgw6r\" (UniqueName: \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\") pod \"maas-api-revoke-sub-34cb65cd-d5w7f\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:31.010965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.010898 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:31.313050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.313027 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f"] Oct 31 00:40:31.315260 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:31.315228 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85793bc6_c88d_461c_a170_9fe82a64e010.slice/crio-6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f WatchSource:0}: Error finding container 6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f: Status 404 returned error can't find the container with id 6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f Oct 31 00:40:31.375276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.375242 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" event={"ID":"f54fab01-0d15-4316-b6b8-370069bd01e9","Type":"ContainerDied","Data":"356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167"} Oct 31 00:40:31.375388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.375296 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356c9b78fb11ed92bbb63d249ed25ad85c294091b6ed81731e049c41a3bf3167" Oct 31 00:40:31.375388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.375257 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w" Oct 31 00:40:31.376535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.376512 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" event={"ID":"85793bc6-c88d-461c-a170-9fe82a64e010","Type":"ContainerStarted","Data":"6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f"} Oct 31 00:40:31.381995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.381979 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" Oct 31 00:40:31.382092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.382026 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m" event={"ID":"4873fe09-d4ba-43e9-b1a3-5db525b3ec69","Type":"ContainerDied","Data":"aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb"} Oct 31 00:40:31.382092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:31.382084 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5d4307631fce236aecb2d4c7534be8aa7f7805cc8e9245edb80612c934cefb" Oct 31 00:40:31.947620 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:40:31.947528 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:40:32.390069 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:32.390039 5011 generic.go:356] "Generic (PLEG): container finished" podID="85793bc6-c88d-461c-a170-9fe82a64e010" containerID="1d87347caba2c3ca267e1349b39d3549279f2723ec7488a5ea37408f9908f935" exitCode=0 Oct 31 00:40:32.390245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:32.390095 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" event={"ID":"85793bc6-c88d-461c-a170-9fe82a64e010","Type":"ContainerDied","Data":"1d87347caba2c3ca267e1349b39d3549279f2723ec7488a5ea37408f9908f935"} Oct 31 00:40:33.543000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.542976 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:33.694560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.694489 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\") pod \"85793bc6-c88d-461c-a170-9fe82a64e010\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " Oct 31 00:40:33.694560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.694535 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\" (UniqueName: \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\") pod \"85793bc6-c88d-461c-a170-9fe82a64e010\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " Oct 31 00:40:33.694791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.694643 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\") pod \"85793bc6-c88d-461c-a170-9fe82a64e010\" (UID: \"85793bc6-c88d-461c-a170-9fe82a64e010\") " Oct 31 00:40:33.694905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.694880 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca" pod "85793bc6-c88d-461c-a170-9fe82a64e010" (UID: "85793bc6-c88d-461c-a170-9fe82a64e010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:33.695042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.695023 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:33.696817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.696791 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca" pod "85793bc6-c88d-461c-a170-9fe82a64e010" (UID: "85793bc6-c88d-461c-a170-9fe82a64e010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:33.696817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.696806 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r" pod "85793bc6-c88d-461c-a170-9fe82a64e010" (UID: "85793bc6-c88d-461c-a170-9fe82a64e010"). InnerVolumeSpecName "kube-api-access-hgw6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:33.795571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.795541 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgw6r\" (UniqueName: \"kubernetes.io/projected/85793bc6-c88d-461c-a170-9fe82a64e010-kube-api-access-hgw6r\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:33.795571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:33.795573 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85793bc6-c88d-461c-a170-9fe82a64e010-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:34.155732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.155695 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c"] Oct 31 00:40:34.156191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.156178 5011 cpu_manager.go:414] "containerMap: removing container" podUID="85793bc6-c88d-461c-a170-9fe82a64e010" containerName="revoke-keys" Oct 31 00:40:34.156252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.156193 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85793bc6-c88d-461c-a170-9fe82a64e010" containerName="revoke-keys" Oct 31 00:40:34.156293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.156279 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="85793bc6-c88d-461c-a170-9fe82a64e010" containerName="revoke-keys" Oct 31 00:40:34.159687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.159665 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.177168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.177145 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c"] Oct 31 00:40:34.299549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.299511 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.299711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.299636 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.299711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.299686 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2nk\" (UniqueName: \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.401166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.401130 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2nk\" (UniqueName: \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.401339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.401284 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.401396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.401361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.402140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.402117 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.403985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.403960 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" Oct 31 00:40:34.404092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.403972 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f" event={"ID":"85793bc6-c88d-461c-a170-9fe82a64e010","Type":"ContainerDied","Data":"6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f"} Oct 31 00:40:34.404092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.404029 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4eccfb51e84cb3ba41ce194ec8d39c97863db27b6bde6d417a908adacc986f" Oct 31 00:40:34.404172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.404091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.423184 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.423164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2nk\" (UniqueName: \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\") pod \"maas-api-revoke-sub-d7494082-sxh6c\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.472879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.472855 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:34.777787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:34.777760 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c"] Oct 31 00:40:34.780989 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:34.780961 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf03f0ce_2c87_48f3_a8d7_4c9eb8d1c29d.slice/crio-549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591 WatchSource:0}: Error finding container 549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591: Status 404 returned error can't find the container with id 549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591 Oct 31 00:40:35.412060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:35.412024 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" event={"ID":"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d","Type":"ContainerStarted","Data":"48f56a315d802faa53d3055ed2d205b262a68a44e71c2cc07031ec058aeb595c"} Oct 31 00:40:35.412253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:35.412082 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" event={"ID":"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d","Type":"ContainerStarted","Data":"549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591"} Oct 31 00:40:35.449300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:35.449254 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" podStartSLOduration=1.449241751 podStartE2EDuration="1.449241751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:35.449058488 +0000 UTC m=+1303.898146127" watchObservedRunningTime="2026-10-31 00:40:35.449241751 +0000 UTC m=+1303.898329384" Oct 31 00:40:36.419916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.419882 5011 generic.go:356] "Generic (PLEG): container finished" podID="af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" containerID="48f56a315d802faa53d3055ed2d205b262a68a44e71c2cc07031ec058aeb595c" exitCode=0 Oct 31 00:40:36.420318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.419962 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" event={"ID":"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d","Type":"ContainerDied","Data":"48f56a315d802faa53d3055ed2d205b262a68a44e71c2cc07031ec058aeb595c"} Oct 31 00:40:36.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.497246 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr"] Oct 31 00:40:36.500938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.500918 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.520194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.520161 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr"] Oct 31 00:40:36.621872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.621841 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.622008 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.621884 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.622064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.622000 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqzpl\" (UniqueName: \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.722695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.722601 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqzpl\" (UniqueName: \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.722846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.722744 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.722846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.722796 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.723592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.723571 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.725299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.725282 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.744672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.744648 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqzpl\" (UniqueName: \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\") pod \"maas-api-revoke-sub-d2e2a215-5ckbr\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:36.817626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:36.817596 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:37.124052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.124027 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr"] Oct 31 00:40:37.127441 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:37.127407 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9442c788_be92_4361_a313_763271f76555.slice/crio-bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62 WatchSource:0}: Error finding container bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62: Status 404 returned error can't find the container with id bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62 Oct 31 00:40:37.430638 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.430604 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" event={"ID":"9442c788-be92-4361-a313-763271f76555","Type":"ContainerStarted","Data":"bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62"} Oct 31 00:40:37.566575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.566548 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:37.632499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.632469 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\" (UniqueName: \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\") pod \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " Oct 31 00:40:37.632655 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.632564 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\") pod \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " Oct 31 00:40:37.632655 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.632625 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\") pod \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\" (UID: \"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d\") " Oct 31 00:40:37.633004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.632975 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca" pod "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" (UID: "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:37.634664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.634637 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk" pod "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" (UID: "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d"). InnerVolumeSpecName "kube-api-access-hc2nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:37.635429 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.635405 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca" pod "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" (UID: "af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:37.733978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.733944 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:37.733978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.733982 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:37.734186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:37.733997 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2nk\" (UniqueName: \"kubernetes.io/projected/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d-kube-api-access-hc2nk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:38.439635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:38.439602 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" Oct 31 00:40:38.439635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:38.439619 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c" event={"ID":"af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d","Type":"ContainerDied","Data":"549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591"} Oct 31 00:40:38.440185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:38.439678 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549f590db574a468083d447d778cc50017c743728073953d017dd95ed8a98591" Oct 31 00:40:38.441803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:38.441772 5011 generic.go:356] "Generic (PLEG): container finished" podID="9442c788-be92-4361-a313-763271f76555" containerID="251430c14a07cc2e07a43059c05833493b19413cf95abeb68e48086fc4d71ae7" exitCode=0 Oct 31 00:40:38.441949 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:38.441879 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" event={"ID":"9442c788-be92-4361-a313-763271f76555","Type":"ContainerDied","Data":"251430c14a07cc2e07a43059c05833493b19413cf95abeb68e48086fc4d71ae7"} Oct 31 00:40:39.605040 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.605007 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:39.755757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.755670 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\") pod \"9442c788-be92-4361-a313-763271f76555\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " Oct 31 00:40:39.755757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.755709 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\" (UniqueName: \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\") pod \"9442c788-be92-4361-a313-763271f76555\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " Oct 31 00:40:39.755989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.755931 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\") pod \"9442c788-be92-4361-a313-763271f76555\" (UID: \"9442c788-be92-4361-a313-763271f76555\") " Oct 31 00:40:39.756097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.756074 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca" pod "9442c788-be92-4361-a313-763271f76555" (UID: "9442c788-be92-4361-a313-763271f76555"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:39.756459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.756437 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9442c788-be92-4361-a313-763271f76555-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:39.758077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.758050 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl" pod "9442c788-be92-4361-a313-763271f76555" (UID: "9442c788-be92-4361-a313-763271f76555"). InnerVolumeSpecName "kube-api-access-tqzpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:39.758567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.758513 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca" pod "9442c788-be92-4361-a313-763271f76555" (UID: "9442c788-be92-4361-a313-763271f76555"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:39.858037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.858002 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqzpl\" (UniqueName: \"kubernetes.io/projected/9442c788-be92-4361-a313-763271f76555-kube-api-access-tqzpl\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:39.858037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.858031 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9442c788-be92-4361-a313-763271f76555-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:39.907560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.907532 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 00:40:39.907985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.907972 5011 cpu_manager.go:414] "containerMap: removing container" podUID="af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" containerName="revoke-keys" Oct 31 00:40:39.908022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.907988 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" containerName="revoke-keys" Oct 31 00:40:39.908022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.908004 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9442c788-be92-4361-a313-763271f76555" containerName="revoke-keys" Oct 31 00:40:39.908022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.908009 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9442c788-be92-4361-a313-763271f76555" containerName="revoke-keys" Oct 31 00:40:39.908128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.908088 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" containerName="revoke-keys" Oct 31 00:40:39.908128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.908100 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9442c788-be92-4361-a313-763271f76555" containerName="revoke-keys" Oct 31 00:40:39.911353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.911333 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:39.926632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.926611 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"distinct-w6-be301b-kserve-self-signed-certs\"" Oct 31 00:40:39.940843 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:39.940816 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 00:40:40.060250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060145 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.060250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060185 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvh7\" (UniqueName: \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.060465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060299 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.060465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060330 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.060554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060487 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.060554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.060508 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161361 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161324 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161377 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161421 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161460 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvh7\" (UniqueName: \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161500 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.161701 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.161543 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.162369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.162336 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.162541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.162510 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.162651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.162567 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.164440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.164415 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.164533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.164493 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.183730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.183707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvh7\" (UniqueName: \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\") pod \"distinct-w6-be301b-kserve-74dc58d6f8-fzfz9\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.224273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.224244 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:40.463237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.460287 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" Oct 31 00:40:40.463237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.460325 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr" event={"ID":"9442c788-be92-4361-a313-763271f76555","Type":"ContainerDied","Data":"bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62"} Oct 31 00:40:40.463237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.460395 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc6bffb4dabbe383fb9ac36ee0699f62a2e50d0cb917af27231d98cbd1e5a62" Oct 31 00:40:40.531791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.531764 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 00:40:40.534250 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:40.534220 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eee9175_5c96_48c2_847d_ffb24b312ba3.slice/crio-eae79885cc919c63f50e2eb7640c9395679b2e98547958f030fbf592e007bdf0 WatchSource:0}: Error finding container eae79885cc919c63f50e2eb7640c9395679b2e98547958f030fbf592e007bdf0: Status 404 returned error can't find the container with id eae79885cc919c63f50e2eb7640c9395679b2e98547958f030fbf592e007bdf0 Oct 31 00:40:40.836161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:40.836127 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:40:41.474670 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:41.474630 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" event={"ID":"9eee9175-5c96-48c2-847d-ffb24b312ba3","Type":"ContainerStarted","Data":"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c"} Oct 31 00:40:41.474670 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:41.474679 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" event={"ID":"9eee9175-5c96-48c2-847d-ffb24b312ba3","Type":"ContainerStarted","Data":"eae79885cc919c63f50e2eb7640c9395679b2e98547958f030fbf592e007bdf0"} Oct 31 00:40:41.516072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:41.516027 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podStartSLOduration=2.516015443 podStartE2EDuration="2.516015443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:41.515355704 +0000 UTC m=+1309.964443344" watchObservedRunningTime="2026-10-31 00:40:41.516015443 +0000 UTC m=+1309.965103081" Oct 31 00:40:42.224853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:42.224821 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:42.968231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:42.947842 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 00:40:42.991230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:42.976321 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.007230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:42.996817 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 00:40:43.007230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.005545 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0d8bae\"/\"distinct-w4-0d8bae-kserve-self-signed-certs\"" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104569 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104633 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104665 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104753 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104779 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.107230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.104853 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mwr\" (UniqueName: \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.209302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.208804 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.209538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.209517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.210067 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.210046 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mwr\" (UniqueName: \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.219950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.219891 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.220470 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.220452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.220975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.220959 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.221495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.209938 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.221624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.209257 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.221710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.221342 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.258229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.255821 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.270233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.268520 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.273247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.271367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mwr\" (UniqueName: \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\") pod \"distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:43.395230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:43.393390 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:44.265610 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:44.261604 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 00:40:44.570240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:44.569292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" event={"ID":"733ad5e7-d7fb-4983-9454-57ac7a885359","Type":"ContainerStarted","Data":"f3d222eb8332c1b60cb700c3ed50ca14a62df72330ed52ca10365bda1384e7f2"} Oct 31 00:40:44.570240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:44.569348 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" event={"ID":"733ad5e7-d7fb-4983-9454-57ac7a885359","Type":"ContainerStarted","Data":"63fda5ec0727e50aa03c89158d3bd9f32c5f16e3f1aa85afe104b815ed77f25c"} Oct 31 00:40:44.656249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:44.655905 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podStartSLOduration=2.655884766 podStartE2EDuration="2.655884766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:44.651739516 +0000 UTC m=+1313.100827155" watchObservedRunningTime="2026-10-31 00:40:44.655884766 +0000 UTC m=+1313.104972405" Oct 31 00:40:45.394781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.394744 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:45.730873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.730798 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4"] Oct 31 00:40:45.734666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.734640 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.753870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.753838 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4"] Oct 31 00:40:45.761089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.761064 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:45.761289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.761268 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:45.835579 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.835547 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.835735 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.835716 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdv89\" (UniqueName: \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.835847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.835827 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.936508 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.936473 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.936665 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.936572 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.936724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.936692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdv89\" (UniqueName: \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.937594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.937564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.939638 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.939612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:45.958977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:45.958950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdv89\" (UniqueName: \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\") pod \"maas-api-revoke-sub-83a70230-k79w4\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:46.048140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:46.048073 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:46.355223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:46.355176 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4"] Oct 31 00:40:46.356432 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:46.356393 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d901f1_e68f_4cc5_9870_d93272905f88.slice/crio-c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13 WatchSource:0}: Error finding container c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13: Status 404 returned error can't find the container with id c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13 Oct 31 00:40:46.584191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:46.584151 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" event={"ID":"20d901f1-e68f-4cc5-9870-d93272905f88","Type":"ContainerStarted","Data":"c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13"} Oct 31 00:40:47.441979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.441899 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:40:47.445683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.445664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.459123 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.459102 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-daaf26-openshift-default-dockercfg-ld79m\"" Oct 31 00:40:47.473364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.473332 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:40:47.554609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554573 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.554747 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.554747 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554643 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.554849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554789 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.554896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554854 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.554940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.554918 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.555052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.555033 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmtf\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.555118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.555104 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.555202 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.555181 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.593805 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.593766 5011 generic.go:356] "Generic (PLEG): container finished" podID="20d901f1-e68f-4cc5-9870-d93272905f88" containerID="8bd7165a8e228df9ef55b93e282b0bec27bb485c966a3e9d369c64026c1ad9b7" exitCode=0 Oct 31 00:40:47.594174 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.593855 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" event={"ID":"20d901f1-e68f-4cc5-9870-d93272905f88","Type":"ContainerDied","Data":"8bd7165a8e228df9ef55b93e282b0bec27bb485c966a3e9d369c64026c1ad9b7"} Oct 31 00:40:47.638004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.637974 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 00:40:47.641841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.641821 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.656684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.656793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656709 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.656793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656733 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.656793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656746 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.656793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656772 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656797 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.656832 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657126 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657265 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657298 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657348 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmtf\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657363 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657491 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657440 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.657571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.657554 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.659047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.659022 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.659418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.659400 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9e377-openshift-default-dockercfg-hqt4g\"" Oct 31 00:40:47.659780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.659760 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.665033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.665013 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 00:40:47.684540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.684512 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.684993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.684957 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmtf\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.758185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758111 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:47.758454 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758433 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758467 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758501 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758592 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllzf\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758625 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758714 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758776 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.758951 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.758929 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.859911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.859883 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.859970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860001 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860027 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllzf\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860154 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860253 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860282 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860381 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860426 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.860803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.860774 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.861096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.861069 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.862818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.862791 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.863235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.863193 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.881722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.881696 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.881948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.881928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllzf\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:47.955810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:47.955787 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:48.086532 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.086505 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:40:48.086953 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:48.086928 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0680007d_233b_4386_bb5c_6f6595f2bdab.slice/crio-4be7de629b32a109686a3dda3030854d21097fccccdbc14feacb5773cb38edc1 WatchSource:0}: Error finding container 4be7de629b32a109686a3dda3030854d21097fccccdbc14feacb5773cb38edc1: Status 404 returned error can't find the container with id 4be7de629b32a109686a3dda3030854d21097fccccdbc14feacb5773cb38edc1 Oct 31 00:40:48.089398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.089365 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.089494 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.089441 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.089494 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.089483 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.268925 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.268896 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 00:40:48.272065 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:40:48.272031 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93750db_7a67_4128_8051_6aa3aa25254c.slice/crio-d19083fec29a51219f285f3d1e78395e26467b393ab6bf6ed80aac07d7c3e8a3 WatchSource:0}: Error finding container d19083fec29a51219f285f3d1e78395e26467b393ab6bf6ed80aac07d7c3e8a3: Status 404 returned error can't find the container with id d19083fec29a51219f285f3d1e78395e26467b393ab6bf6ed80aac07d7c3e8a3 Oct 31 00:40:48.274649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.274616 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.274750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.274691 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.274750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.274740 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:40:48.609523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.608821 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" event={"ID":"b93750db-7a67-4128-8051-6aa3aa25254c","Type":"ContainerStarted","Data":"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70"} Oct 31 00:40:48.609523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.608890 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" event={"ID":"b93750db-7a67-4128-8051-6aa3aa25254c","Type":"ContainerStarted","Data":"d19083fec29a51219f285f3d1e78395e26467b393ab6bf6ed80aac07d7c3e8a3"} Oct 31 00:40:48.611783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.611710 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" event={"ID":"0680007d-233b-4386-bb5c-6f6595f2bdab","Type":"ContainerStarted","Data":"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd"} Oct 31 00:40:48.611783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.611764 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" event={"ID":"0680007d-233b-4386-bb5c-6f6595f2bdab","Type":"ContainerStarted","Data":"4be7de629b32a109686a3dda3030854d21097fccccdbc14feacb5773cb38edc1"} Oct 31 00:40:48.662176 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.658287 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podStartSLOduration=1.658268555 podStartE2EDuration="1.658268555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:48.655775761 +0000 UTC m=+1317.104863403" watchObservedRunningTime="2026-10-31 00:40:48.658268555 +0000 UTC m=+1317.107356196" Oct 31 00:40:48.762835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.762789 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:48.864294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.862775 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:48.887075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.886249 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\" (UniqueName: \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\") pod \"20d901f1-e68f-4cc5-9870-d93272905f88\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " Oct 31 00:40:48.887075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.886412 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\") pod \"20d901f1-e68f-4cc5-9870-d93272905f88\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " Oct 31 00:40:48.887075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.886463 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\") pod \"20d901f1-e68f-4cc5-9870-d93272905f88\" (UID: \"20d901f1-e68f-4cc5-9870-d93272905f88\") " Oct 31 00:40:48.887674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.887623 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca" pod "20d901f1-e68f-4cc5-9870-d93272905f88" (UID: "20d901f1-e68f-4cc5-9870-d93272905f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:48.888892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.888868 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca" pod "20d901f1-e68f-4cc5-9870-d93272905f88" (UID: "20d901f1-e68f-4cc5-9870-d93272905f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:48.889944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.889923 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89" pod "20d901f1-e68f-4cc5-9870-d93272905f88" (UID: "20d901f1-e68f-4cc5-9870-d93272905f88"). InnerVolumeSpecName "kube-api-access-sdv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:48.929757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.929677 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podStartSLOduration=1.929657425 podStartE2EDuration="1.929657425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:48.706693833 +0000 UTC m=+1317.155781473" watchObservedRunningTime="2026-10-31 00:40:48.929657425 +0000 UTC m=+1317.378745065" Oct 31 00:40:48.988076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.988036 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdv89\" (UniqueName: \"kubernetes.io/projected/20d901f1-e68f-4cc5-9870-d93272905f88-kube-api-access-sdv89\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:48.988076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.988077 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:48.988347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.988092 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d901f1-e68f-4cc5-9870-d93272905f88-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:48.988347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:48.988155 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:49.413157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.413121 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn"] Oct 31 00:40:49.413804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.413782 5011 cpu_manager.go:414] "containerMap: removing container" podUID="20d901f1-e68f-4cc5-9870-d93272905f88" containerName="revoke-keys" Oct 31 00:40:49.413804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.413800 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d901f1-e68f-4cc5-9870-d93272905f88" containerName="revoke-keys" Oct 31 00:40:49.413938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.413903 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d901f1-e68f-4cc5-9870-d93272905f88" containerName="revoke-keys" Oct 31 00:40:49.417253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.417229 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.484638 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.479256 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn"] Oct 31 00:40:49.493237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.492951 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.493237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.493042 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.493237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.493154 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fx9\" (UniqueName: \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.594188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.594152 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.594377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.594224 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fx9\" (UniqueName: \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.594377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.594309 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.595357 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.595332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.598290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.598268 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.617604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.617579 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fx9\" (UniqueName: \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\") pod \"maas-api-revoke-sub-6e828089-fjtsn\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.649241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.639671 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" Oct 31 00:40:49.658141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.640289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4" event={"ID":"20d901f1-e68f-4cc5-9870-d93272905f88","Type":"ContainerDied","Data":"c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13"} Oct 31 00:40:49.658409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.658386 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ba749f266f10c100aeac5f5eb5aceafdfb151d5b269ea23d2641f0819cdd13" Oct 31 00:40:49.761660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.761339 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:40:49.761660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.761404 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 00:40:49.784853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.784713 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:49.976793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.976699 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:40:49.976793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:49.976777 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 00:40:50.174229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.172241 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn"] Oct 31 00:40:50.244284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.230606 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:50.279500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.279473 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:50.703079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.703047 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" event={"ID":"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5","Type":"ContainerStarted","Data":"405d1d1d05994304371beb06ed963b191d6c6e8457ee37ac680bf40c013fbc3e"} Oct 31 00:40:50.780231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.779551 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:40:50.780231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.779596 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 00:40:50.780231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.779707 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:40:50.973228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.966287 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:40:50.973228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:50.966368 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 00:40:51.711292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.711202 5011 generic.go:356] "Generic (PLEG): container finished" podID="ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" containerID="df6fb815097ccd5f75fe56ce0d8315f3cf181d4b7d66df414eaa76de39861314" exitCode=0 Oct 31 00:40:51.711728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.711237 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" event={"ID":"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5","Type":"ContainerDied","Data":"df6fb815097ccd5f75fe56ce0d8315f3cf181d4b7d66df414eaa76de39861314"} Oct 31 00:40:51.765231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.764798 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:40:51.765231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.764853 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 00:40:51.959643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.959607 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:40:51.959933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:51.959898 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 00:40:52.758980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:52.758546 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:40:52.758980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:52.758607 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 00:40:52.984286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:52.979834 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:40:52.984286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:52.979906 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 00:40:53.085317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.085291 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:53.187432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.187392 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\" (UniqueName: \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\") pod \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " Oct 31 00:40:53.187626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.187465 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\") pod \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " Oct 31 00:40:53.187626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.187574 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\") pod \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\" (UID: \"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5\") " Oct 31 00:40:53.188136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.188112 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca" pod "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" (UID: "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:53.190241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.190147 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9" pod "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" (UID: "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5"). InnerVolumeSpecName "kube-api-access-l4fx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:53.190241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.190159 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca" pod "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" (UID: "ad56d3cb-93a4-41ed-b2c5-906e936bd1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:53.289373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.289279 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:53.289373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.289311 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fx9\" (UniqueName: \"kubernetes.io/projected/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-kube-api-access-l4fx9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:53.289373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.289321 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:40:53.394521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.394487 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:53.407818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.407788 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:53.772230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.771751 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" Oct 31 00:40:53.772721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.772310 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn" event={"ID":"ad56d3cb-93a4-41ed-b2c5-906e936bd1e5","Type":"ContainerDied","Data":"405d1d1d05994304371beb06ed963b191d6c6e8457ee37ac680bf40c013fbc3e"} Oct 31 00:40:53.772721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.772351 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405d1d1d05994304371beb06ed963b191d6c6e8457ee37ac680bf40c013fbc3e" Oct 31 00:40:53.776229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.774065 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:40:53.776229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.774123 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 00:40:53.795888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.795862 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:40:53.972451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.969332 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:40:53.972451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:53.969400 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 00:40:54.933029 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:54.932994 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:54.934441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:54.934349 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:54.938423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:54.938398 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:40:55.264897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:55.264803 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:55.266171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:55.266145 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:40:55.267169 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:40:55.267137 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 00:41:00.454939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.454900 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c"] Oct 31 00:41:00.455672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.455642 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" containerName="revoke-keys" Oct 31 00:41:00.455778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.455673 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" containerName="revoke-keys" Oct 31 00:41:00.455874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.455857 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" containerName="revoke-keys" Oct 31 00:41:00.496231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.490568 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c"] Oct 31 00:41:00.496231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.490691 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.505537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.504596 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:41:00.515102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.515072 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:41:00.598845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.598814 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.599065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.599051 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.599140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.599129 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfm9\" (UniqueName: \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.700811 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.700776 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.701877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.701854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfm9\" (UniqueName: \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.702549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.702524 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.703009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.701784 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.714929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.714605 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.727879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.727828 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfm9\" (UniqueName: \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\") pod \"maas-api-revoke-sub-f9c7fb04-bdw6c\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:00.818730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:00.818695 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:01.191540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:01.191508 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c"] Oct 31 00:41:01.196640 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:01.196607 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb50d93_ed63_4341_a950_3ce28473742b.slice/crio-b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f WatchSource:0}: Error finding container b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f: Status 404 returned error can't find the container with id b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f Oct 31 00:41:01.837007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:01.836967 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" event={"ID":"bcb50d93-ed63-4341-a950-3ce28473742b","Type":"ContainerStarted","Data":"4622f5d5cae2eb3432619914e06a9d98347505b70f7f0625da98e33ae00e85d6"} Oct 31 00:41:01.837495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:01.837035 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" event={"ID":"bcb50d93-ed63-4341-a950-3ce28473742b","Type":"ContainerStarted","Data":"b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f"} Oct 31 00:41:01.893428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:01.893373 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" podStartSLOduration=1.893359816 podStartE2EDuration="1.893359816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:01.891625951 +0000 UTC m=+1330.340713591" watchObservedRunningTime="2026-10-31 00:41:01.893359816 +0000 UTC m=+1330.342447454" Oct 31 00:41:01.947153 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:41:01.947115 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:41:02.847080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:02.847041 5011 generic.go:356] "Generic (PLEG): container finished" podID="bcb50d93-ed63-4341-a950-3ce28473742b" containerID="4622f5d5cae2eb3432619914e06a9d98347505b70f7f0625da98e33ae00e85d6" exitCode=0 Oct 31 00:41:02.847494 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:02.847121 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" event={"ID":"bcb50d93-ed63-4341-a950-3ce28473742b","Type":"ContainerDied","Data":"4622f5d5cae2eb3432619914e06a9d98347505b70f7f0625da98e33ae00e85d6"} Oct 31 00:41:03.187935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.187857 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 00:41:03.191759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.191740 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.223005 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.222984 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0d8bae-openshift-default-dockercfg-bvb9r\"" Oct 31 00:41:03.234534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234507 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234544 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db226\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234627 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234681 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234700 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234789 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234815 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234855 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.234961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.234947 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.245004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.244975 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 00:41:03.335538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335506 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335539 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db226\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335574 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335703 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335735 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335767 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335787 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335831 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.335931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335874 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.336142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.335948 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.336142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.336129 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.336268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.336241 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.336405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.336386 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.336623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.336603 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.338180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.338157 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.338375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.338357 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.357173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.357139 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.357348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.357330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db226\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.505257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.505159 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:03.826825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.826799 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 00:41:03.829036 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:03.828999 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3803ca5d_c7cf_4aad_a5ef_debcce4ed7e5.slice/crio-f8386ffa6c383948495685625705b81cfa93e6c5203cd1132f8c7dfae69822cf WatchSource:0}: Error finding container f8386ffa6c383948495685625705b81cfa93e6c5203cd1132f8c7dfae69822cf: Status 404 returned error can't find the container with id f8386ffa6c383948495685625705b81cfa93e6c5203cd1132f8c7dfae69822cf Oct 31 00:41:03.832428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.832035 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:03.832428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.832103 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:03.832428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.832138 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:03.855462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:03.855436 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" event={"ID":"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5","Type":"ContainerStarted","Data":"f8386ffa6c383948495685625705b81cfa93e6c5203cd1132f8c7dfae69822cf"} Oct 31 00:41:04.042582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.042553 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:04.146721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.146622 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\" (UniqueName: \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\") pod \"bcb50d93-ed63-4341-a950-3ce28473742b\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " Oct 31 00:41:04.146887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.146746 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\") pod \"bcb50d93-ed63-4341-a950-3ce28473742b\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " Oct 31 00:41:04.146887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.146787 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\") pod \"bcb50d93-ed63-4341-a950-3ce28473742b\" (UID: \"bcb50d93-ed63-4341-a950-3ce28473742b\") " Oct 31 00:41:04.147443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.147391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca" pod "bcb50d93-ed63-4341-a950-3ce28473742b" (UID: "bcb50d93-ed63-4341-a950-3ce28473742b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:04.149307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.149284 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca" pod "bcb50d93-ed63-4341-a950-3ce28473742b" (UID: "bcb50d93-ed63-4341-a950-3ce28473742b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:04.149724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.149706 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9" pod "bcb50d93-ed63-4341-a950-3ce28473742b" (UID: "bcb50d93-ed63-4341-a950-3ce28473742b"). InnerVolumeSpecName "kube-api-access-qdfm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:04.248490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.248000 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfm9\" (UniqueName: \"kubernetes.io/projected/bcb50d93-ed63-4341-a950-3ce28473742b-kube-api-access-qdfm9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:41:04.248490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.248036 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:41:04.248490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.248049 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb50d93-ed63-4341-a950-3ce28473742b-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:41:04.868369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.868332 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" event={"ID":"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5","Type":"ContainerStarted","Data":"046b85f6c9a41b548aec02bd600502e1116c0228641037e4dc085e564b17598d"} Oct 31 00:41:04.870602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.870575 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" event={"ID":"bcb50d93-ed63-4341-a950-3ce28473742b","Type":"ContainerDied","Data":"b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f"} Oct 31 00:41:04.870761 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.870614 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a8cab7d3993c649db113b977e9be5c9f804b57b6a844a0eb72639722583f3f" Oct 31 00:41:04.870761 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.870660 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c" Oct 31 00:41:04.944809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:04.943427 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podStartSLOduration=1.9434081189999999 podStartE2EDuration="1.943408119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:04.938069304 +0000 UTC m=+1333.387156973" watchObservedRunningTime="2026-10-31 00:41:04.943408119 +0000 UTC m=+1333.392495760" Oct 31 00:41:05.506025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:05.505988 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:05.507793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:05.507767 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:05.507908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:05.507815 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:06.508024 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:06.507978 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:06.508597 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:06.508056 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:07.506136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:07.506100 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:07.506340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:07.506156 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:08.051388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:08.051351 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:41:08.506461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:08.506425 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:08.506627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:08.506484 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:09.506563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:09.506531 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:09.506940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:09.506582 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:10.506360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:10.506324 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:10.506538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:10.506389 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:41:11.510101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:11.510066 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:11.510562 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:11.510505 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:11.511133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:11.511114 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 00:41:17.464829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.464798 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 00:41:17.465316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.465301 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bcb50d93-ed63-4341-a950-3ce28473742b" containerName="revoke-keys" Oct 31 00:41:17.465316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.465316 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb50d93-ed63-4341-a950-3ce28473742b" containerName="revoke-keys" Oct 31 00:41:17.465430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.465420 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb50d93-ed63-4341-a950-3ce28473742b" containerName="revoke-keys" Oct 31 00:41:17.468891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.468870 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.485464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.485445 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-735123-openshift-default-dockercfg-z4chw\"" Oct 31 00:41:17.498477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.498449 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 00:41:17.543633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543659 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543722 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543762 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543779 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543880 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543899 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8l9w\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.543933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543926 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.544002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.543963 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645084 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8l9w\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645255 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645321 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645404 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645527 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.645878 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645856 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.646018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.645951 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.646249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.646200 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.647661 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.647638 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.647937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.647920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.666835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.666810 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.667042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.667021 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8l9w\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.781781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.781661 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:17.859350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.859318 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:41:17.865233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.865060 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.878977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.878692 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-671959-openshift-default-dockercfg-mgz7q\"" Oct 31 00:41:17.892839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.892814 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:41:17.949382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949354 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949389 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949417 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949575 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949681 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949708 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbq8\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949736 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:17.949885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:17.949872 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051081 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051119 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbq8\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051151 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051251 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051283 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051311 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051343 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051491 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051491 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051425 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.051791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051725 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.052002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.051977 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.052106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.052044 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.052106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.052067 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.052415 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.052393 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.054058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.054035 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.054273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.054256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.075241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.075221 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbq8\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.079039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.079020 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.108765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.108743 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 00:41:18.111434 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:18.111408 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaaf1adf_a248_461c_ab68_f7743527db00.slice/crio-0eb778f47ab310404d05312c90bcc026173708de4fc44aa516b7f13b7427ea82 WatchSource:0}: Error finding container 0eb778f47ab310404d05312c90bcc026173708de4fc44aa516b7f13b7427ea82: Status 404 returned error can't find the container with id 0eb778f47ab310404d05312c90bcc026173708de4fc44aa516b7f13b7427ea82 Oct 31 00:41:18.114293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.114253 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.114396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.114341 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.114396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.114385 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.188769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.188747 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:18.548561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.548526 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:41:18.554409 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:18.554383 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ae3679_1fd1_44eb_bf75_568e859767c8.slice/crio-2ffcf5dc6cf311a66899b9bf5a83ed8883491342a56b79f8b30811633c05376c WatchSource:0}: Error finding container 2ffcf5dc6cf311a66899b9bf5a83ed8883491342a56b79f8b30811633c05376c: Status 404 returned error can't find the container with id 2ffcf5dc6cf311a66899b9bf5a83ed8883491342a56b79f8b30811633c05376c Oct 31 00:41:18.557031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.556993 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.557106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.557079 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.557145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.557121 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:18.980508 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.980470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" event={"ID":"daaf1adf-a248-461c-ab68-f7743527db00","Type":"ContainerStarted","Data":"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1"} Oct 31 00:41:18.980699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.980526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" event={"ID":"daaf1adf-a248-461c-ab68-f7743527db00","Type":"ContainerStarted","Data":"0eb778f47ab310404d05312c90bcc026173708de4fc44aa516b7f13b7427ea82"} Oct 31 00:41:18.982514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.982486 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" event={"ID":"24ae3679-1fd1-44eb-bf75-568e859767c8","Type":"ContainerStarted","Data":"c228936e21121cc0fc5404ab92f095eb042bcecc08b37f9b00668c3ce1eae446"} Oct 31 00:41:18.982640 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:18.982530 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" event={"ID":"24ae3679-1fd1-44eb-bf75-568e859767c8","Type":"ContainerStarted","Data":"2ffcf5dc6cf311a66899b9bf5a83ed8883491342a56b79f8b30811633c05376c"} Oct 31 00:41:19.087586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.087533 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podStartSLOduration=2.087517756 podStartE2EDuration="2.087517756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.085131563 +0000 UTC m=+1347.534219203" watchObservedRunningTime="2026-10-31 00:41:19.087517756 +0000 UTC m=+1347.536605395" Oct 31 00:41:19.089050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.089007 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podStartSLOduration=2.088998518 podStartE2EDuration="2.088998518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.035554859 +0000 UTC m=+1347.484642502" watchObservedRunningTime="2026-10-31 00:41:19.088998518 +0000 UTC m=+1347.538086161" Oct 31 00:41:19.189485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.189448 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:19.191116 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.191088 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:19.191279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.191145 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:19.782232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.782096 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:19.783874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.783846 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:19.784102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:19.784074 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:20.189643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:20.189607 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:20.189823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:20.189670 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:20.784230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:20.784116 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:20.784230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:20.784175 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:21.190231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:21.189589 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:21.190231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:21.189654 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:21.785637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:21.785601 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:21.786176 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:21.786141 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:22.189083 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:22.189047 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:22.189339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:22.189123 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:22.784877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:22.784530 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:22.784877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:22.784584 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:23.189989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:23.189947 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:23.190510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:23.190016 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:23.783855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:23.783165 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:23.783855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:23.783245 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:24.190187 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:24.190145 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:24.190786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:24.190241 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:24.783898 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:24.783860 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:41:24.784064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:24.783925 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 00:41:25.189243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:25.189198 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:41:25.189418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:25.189267 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:41:25.788066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:25.788033 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:25.788613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:25.788409 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:25.789095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:25.789077 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 00:41:26.208236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:26.208177 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:26.208580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:26.208547 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:26.209602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:26.209582 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:41:28.806403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.806363 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 00:41:28.812792 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.812765 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.843564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.843534 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"distinct-2-w6-be301b-kserve-self-signed-certs\"" Oct 31 00:41:28.846887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.846863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 00:41:28.859288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859254 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.859410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859363 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.859410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859389 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcj5\" (UniqueName: \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.859531 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859512 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.859593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859576 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.859658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.859644 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960440 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960476 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcj5\" (UniqueName: \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960531 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960579 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960644 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960697 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.960979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.960951 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.961052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.961031 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.961258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.961238 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.963556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.963532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.963766 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.963747 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:28.983093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:28.983063 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcj5\" (UniqueName: \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\") pod \"distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:29.130017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:29.129979 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:29.441383 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:29.441357 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 00:41:29.443380 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:29.443347 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cefe0ca_326c_4a75_b158_60ce35efaa4f.slice/crio-0e709e99121fd7dec4855857e66ed79ade9a7d0404d297e3f4bbeaa2bf8026ae WatchSource:0}: Error finding container 0e709e99121fd7dec4855857e66ed79ade9a7d0404d297e3f4bbeaa2bf8026ae: Status 404 returned error can't find the container with id 0e709e99121fd7dec4855857e66ed79ade9a7d0404d297e3f4bbeaa2bf8026ae Oct 31 00:41:30.088104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:30.088067 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" event={"ID":"4cefe0ca-326c-4a75-b158-60ce35efaa4f","Type":"ContainerStarted","Data":"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675"} Oct 31 00:41:30.088502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:30.088119 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" event={"ID":"4cefe0ca-326c-4a75-b158-60ce35efaa4f","Type":"ContainerStarted","Data":"0e709e99121fd7dec4855857e66ed79ade9a7d0404d297e3f4bbeaa2bf8026ae"} Oct 31 00:41:30.130918 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:30.130888 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:30.131098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:30.130990 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podStartSLOduration=2.130974202 podStartE2EDuration="2.130974202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:30.129892363 +0000 UTC m=+1358.578980002" watchObservedRunningTime="2026-10-31 00:41:30.130974202 +0000 UTC m=+1358.580061843" Oct 31 00:41:31.953810 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:41:31.953766 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:41:32.901825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:32.901788 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 00:41:32.905987 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:32.905959 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:32.929026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:32.928996 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 00:41:33.009250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009216 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009275 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009322 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwmf\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009413 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009540 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009591 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009692 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.009941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.009733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.110911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.110880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.110911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.110923 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111047 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111080 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111368 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111463 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111499 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111592 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwmf\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111670 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111845 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.111987 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.111962 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.112143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.112127 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.113458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.113431 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.113870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.113854 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.136570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.136540 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.137997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.137971 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwmf\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.220200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.220139 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:33.548889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.548857 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 00:41:33.549306 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:33.549282 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c1d831_fefc_4d9e_9fc9_107235d04648.slice/crio-3d38c00ba420a4d7d80fec14c77a9e9be9a4d0ace36103bc7b4be61625fc8797 WatchSource:0}: Error finding container 3d38c00ba420a4d7d80fec14c77a9e9be9a4d0ace36103bc7b4be61625fc8797: Status 404 returned error can't find the container with id 3d38c00ba420a4d7d80fec14c77a9e9be9a4d0ace36103bc7b4be61625fc8797 Oct 31 00:41:33.552064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.552024 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:33.552324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.552112 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:33.552324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:33.552157 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:34.117418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.117375 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" event={"ID":"d0c1d831-fefc-4d9e-9fc9-107235d04648","Type":"ContainerStarted","Data":"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a"} Oct 31 00:41:34.117872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.117445 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" event={"ID":"d0c1d831-fefc-4d9e-9fc9-107235d04648","Type":"ContainerStarted","Data":"3d38c00ba420a4d7d80fec14c77a9e9be9a4d0ace36103bc7b4be61625fc8797"} Oct 31 00:41:34.163081 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.163021 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podStartSLOduration=2.163005321 podStartE2EDuration="2.163005321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:34.15926365 +0000 UTC m=+1362.608351288" watchObservedRunningTime="2026-10-31 00:41:34.163005321 +0000 UTC m=+1362.612092959" Oct 31 00:41:34.220327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.220296 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:34.223709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.223681 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:34.223863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:34.223737 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:35.223607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:35.223567 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:35.224198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:35.223632 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:36.222201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:36.222167 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:36.222502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:36.222457 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:37.099354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.099319 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:41:37.221570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.221534 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:37.221827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.221799 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:37.654364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.654294 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 00:41:37.660703 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.660664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.672863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.672659 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 00:41:37.700239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.697028 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"distinct-w3-daaf26-kserve-self-signed-certs\"" Oct 31 00:41:37.780947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.780915 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.781132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.781061 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.781132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.781096 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.781132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.781124 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.781364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.781341 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmvf\" (UniqueName: \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.781425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.781377 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.882504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882468 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmvf\" (UniqueName: \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.882504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882508 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.882773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882538 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.882773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882670 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.882773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882699 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.883023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.882999 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.883101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.883045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.883193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.883161 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.883305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.883289 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.889233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.885283 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.889233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.885287 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.946411 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.946326 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmvf\" (UniqueName: \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\") pod \"distinct-w3-daaf26-kserve-56f479c94-l7wb5\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:37.979432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:37.979402 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:38.242027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:38.241534 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:38.242027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:38.241594 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:38.407274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:38.407231 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 00:41:38.464805 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:38.464749 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13cb12cf_6050_4d2a_be4b_01a159d5a3c6.slice/crio-2052b81022a2f30534ba03acfe9ba5afac9c426d50a06d101c62a3e68afd0edb WatchSource:0}: Error finding container 2052b81022a2f30534ba03acfe9ba5afac9c426d50a06d101c62a3e68afd0edb: Status 404 returned error can't find the container with id 2052b81022a2f30534ba03acfe9ba5afac9c426d50a06d101c62a3e68afd0edb Oct 31 00:41:39.133426 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.132097 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:39.180563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.180521 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:39.273557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.273508 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:39.274160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.274126 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:39.288276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.288242 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" event={"ID":"13cb12cf-6050-4d2a-be4b-01a159d5a3c6","Type":"ContainerStarted","Data":"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e"} Oct 31 00:41:39.288454 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.288435 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" event={"ID":"13cb12cf-6050-4d2a-be4b-01a159d5a3c6","Type":"ContainerStarted","Data":"2052b81022a2f30534ba03acfe9ba5afac9c426d50a06d101c62a3e68afd0edb"} Oct 31 00:41:39.304623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.304594 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:41:39.358231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.357485 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podStartSLOduration=2.357466822 podStartE2EDuration="2.357466822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:39.340085772 +0000 UTC m=+1367.789173412" watchObservedRunningTime="2026-10-31 00:41:39.357466822 +0000 UTC m=+1367.806554461" Oct 31 00:41:39.980498 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:39.980465 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:40.220733 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:40.220693 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:41:40.220909 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:40.220774 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 00:41:41.225884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:41.225853 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:41.226440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:41.226415 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:41.227237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:41.227199 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 00:41:47.280161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.280121 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:41:47.284439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.284421 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.299455 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.299436 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b024b5-openshift-default-dockercfg-gwjft\"" Oct 31 00:41:47.315134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.315105 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:41:47.362070 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362038 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362070 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362072 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn74t\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362104 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362289 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362352 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362428 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362498 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362480 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362510 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.362545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.362531 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.463806 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.463780 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.463970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.463813 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.463970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.463833 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.463970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.463871 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.463977 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn74t\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464029 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464128 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464219 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464234 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464278 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464309 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464487 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464684 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.464783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.464684 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.466256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.466231 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.466536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.466517 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.486439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.486415 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.486640 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.486620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn74t\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.598965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.598943 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:47.824462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.824424 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:41:47.829540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.829514 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.855295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.855223 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:41:47.869413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.869642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869603 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.869756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869716 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhhh\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.869824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869766 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.869824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869806 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.869972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.869950 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.870254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.870114 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.870254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.870241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.870434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.870303 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.923275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.923250 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:41:47.926009 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:41:47.925986 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4855c225_bec3_4f4a_a815_dbe5ba42b902.slice/crio-b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b WatchSource:0}: Error finding container b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b: Status 404 returned error can't find the container with id b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b Oct 31 00:41:47.928378 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.928348 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:47.928469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.928415 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:47.928469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.928445 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:47.971656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971632 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971664 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhhh\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971714 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971744 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971804 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971843 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971895 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.971995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.971940 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.972294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.972100 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.972435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.972383 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.972594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.972485 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.972594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.972508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.972858 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.972795 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.974321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.974301 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.974669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.974651 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.980615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.980595 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:47.993673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.993651 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:47.995924 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.995899 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhhh\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:47.996390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:47.996367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:48.144444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.144365 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:48.358037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.357999 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" event={"ID":"4855c225-bec3-4f4a-a815-dbe5ba42b902","Type":"ContainerStarted","Data":"39b29288e9b2336d949dc9d94bacf9a8645641e3655b1f8aefe91d5051a7dd46"} Oct 31 00:41:48.358554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.358064 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" event={"ID":"4855c225-bec3-4f4a-a815-dbe5ba42b902","Type":"ContainerStarted","Data":"b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b"} Oct 31 00:41:48.372543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.372518 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:41:48.404502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.404382 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podStartSLOduration=1.404367272 podStartE2EDuration="1.404367272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:48.402375294 +0000 UTC m=+1376.851462958" watchObservedRunningTime="2026-10-31 00:41:48.404367272 +0000 UTC m=+1376.853454910" Oct 31 00:41:48.472454 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.472419 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:41:48.481058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.481026 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:48.481162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.481115 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:48.481162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.481156 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:41:48.599441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.599406 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:48.600803 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.600774 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:48.600913 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:48.600827 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:49.386154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:49.386123 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" event={"ID":"08051a91-99a7-4823-8150-24e545e2e4f3","Type":"ContainerStarted","Data":"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab"} Oct 31 00:41:49.386563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:49.386201 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" event={"ID":"08051a91-99a7-4823-8150-24e545e2e4f3","Type":"ContainerStarted","Data":"d9a79ccdb5a518c77807e5b4855ba8b5af9bc847906bd4ea406e573c7f24960f"} Oct 31 00:41:49.463230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:49.454682 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podStartSLOduration=2.454658941 podStartE2EDuration="2.454658941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:49.440637853 +0000 UTC m=+1377.889725494" watchObservedRunningTime="2026-10-31 00:41:49.454658941 +0000 UTC m=+1377.903746581" Oct 31 00:41:49.600269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:49.600237 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:49.600438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:49.600289 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:50.155438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:50.155403 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:50.157844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:50.157813 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:50.158063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:50.158035 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:41:50.599534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:50.599496 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:50.599906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:50.599550 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:51.148655 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:51.148615 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:51.148953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:51.148922 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:41:51.599740 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:51.599709 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:51.600192 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:51.599771 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:52.147675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:52.147632 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:52.147932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:52.147902 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:41:52.599289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:52.599260 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:52.599551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:52.599530 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:53.147619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:53.147579 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:53.148265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:53.148196 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:41:53.600315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:53.600285 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:53.600535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:53.600511 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:54.145600 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:54.145560 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:54.145888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:54.145859 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:41:54.599352 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:54.599311 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:54.599759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:54.599379 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 00:41:55.410645 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.410494 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:55.411804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.411777 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:55.412666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.412648 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:41:55.604340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.604314 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:55.604796 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.604715 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:41:55.605435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:41:55.605420 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:42:01.991771 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:42:01.991680 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:42:02.647694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.647660 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:42:02.652870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.652844 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.680181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.680145 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:42:02.711767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.711734 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnxh\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.711937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.711786 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.711937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.711803 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.711937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.711891 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.712120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.711991 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.712120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.712052 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.712249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.712141 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.712249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.712180 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.712370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.712310 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813548 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813511 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813548 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813547 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813581 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813619 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813658 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813729 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.813826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813764 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813849 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813976 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnxh\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.813986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.814328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.814588 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.814860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.814794 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.815872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.815843 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.817513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.817484 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.817667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.817650 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.835678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.835642 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.836045 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.835999 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnxh\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:02.968883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:02.968773 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:03.133555 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.133518 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 00:42:03.138683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.138662 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.158107 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.158072 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be301b-openshift-default-dockercfg-4bp8d\"" Oct 31 00:42:03.162925 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.162897 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 00:42:03.218347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218312 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218527 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218354 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218518 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218582 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218664 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218698 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218735 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218761 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.218855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.218809 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.312019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.311986 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:42:03.319931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.319902 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.319947 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320018 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320122 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320262 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320400 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.320562 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.320532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.321261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.321175 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.321523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.321497 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.321596 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.321510 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.321698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.321603 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.322551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.322532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.323460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.323441 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.324970 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:03.324946 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ac50da_c78d_489a_9fa1_686293c3ac84.slice/crio-0f092e69ab2e6f3d59b8f74ecbc5bf67e105995234526da7294dac640ca796bf WatchSource:0}: Error finding container 0f092e69ab2e6f3d59b8f74ecbc5bf67e105995234526da7294dac640ca796bf: Status 404 returned error can't find the container with id 0f092e69ab2e6f3d59b8f74ecbc5bf67e105995234526da7294dac640ca796bf Oct 31 00:42:03.328036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.327996 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.328130 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.328080 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.328237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.328136 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.342073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.342052 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.342526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.342508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.453838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.453807 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:03.506978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.506893 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" event={"ID":"18ac50da-c78d-489a-9fa1-686293c3ac84","Type":"ContainerStarted","Data":"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19"} Oct 31 00:42:03.506978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.506962 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" event={"ID":"18ac50da-c78d-489a-9fa1-686293c3ac84","Type":"ContainerStarted","Data":"0f092e69ab2e6f3d59b8f74ecbc5bf67e105995234526da7294dac640ca796bf"} Oct 31 00:42:03.556223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.555970 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podStartSLOduration=1.555955846 podStartE2EDuration="1.555955846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:03.553062025 +0000 UTC m=+1392.002149668" watchObservedRunningTime="2026-10-31 00:42:03.555955846 +0000 UTC m=+1392.005043485" Oct 31 00:42:03.820458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.819987 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 00:42:03.844267 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:03.823089 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ccd2b_c57d_4fc7_882d_e55eebbf2092.slice/crio-81c39a7c8646c71823495411b2201ffc733596e4dfdfe15328489dff30e53281 WatchSource:0}: Error finding container 81c39a7c8646c71823495411b2201ffc733596e4dfdfe15328489dff30e53281: Status 404 returned error can't find the container with id 81c39a7c8646c71823495411b2201ffc733596e4dfdfe15328489dff30e53281 Oct 31 00:42:03.844267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.826470 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.844267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.826538 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.844267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.826568 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:03.969250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:03.968941 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:04.520234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:04.519997 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" event={"ID":"661ccd2b-c57d-4fc7-882d-e55eebbf2092","Type":"ContainerStarted","Data":"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499"} Oct 31 00:42:04.520234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:04.520056 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" event={"ID":"661ccd2b-c57d-4fc7-882d-e55eebbf2092","Type":"ContainerStarted","Data":"81c39a7c8646c71823495411b2201ffc733596e4dfdfe15328489dff30e53281"} Oct 31 00:42:04.577231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:04.572628 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podStartSLOduration=1.572605995 podStartE2EDuration="1.572605995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:04.567802679 +0000 UTC m=+1393.016890352" watchObservedRunningTime="2026-10-31 00:42:04.572605995 +0000 UTC m=+1393.021693634" Oct 31 00:42:04.972382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:04.972345 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:04.972664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:04.972633 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:05.462231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:05.455386 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:05.462231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:05.457263 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:05.462231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:05.457403 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:05.971526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:05.971495 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:05.971994 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:05.971969 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:06.454617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:06.454579 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:06.454894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:06.454862 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:06.987232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:06.974312 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:06.987232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:06.974376 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:07.455327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:07.455288 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:07.455607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:07.455578 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:07.974232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:07.973715 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:07.974232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:07.973789 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:08.460231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:08.455545 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:08.460231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:08.455608 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:08.969970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:08.969553 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:08.969970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:08.969611 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:09.456565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:09.456527 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:09.456861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:09.456833 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:09.970121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:09.970087 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:42:09.970624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:09.970587 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:42:10.454235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:10.454176 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:42:10.454546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:10.454494 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:42:10.986319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:10.986279 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:10.987276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:10.987244 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:10.988852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:10.988822 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:42:11.673573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:11.673532 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:11.673973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:11.673938 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:11.684766 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:11.684739 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 00:42:15.071334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:15.071295 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:42:17.701971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.701938 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:42:17.709285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.709256 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.736127 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.735240 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:42:17.762402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.761994 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 00:42:17.771252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.770448 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.797287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797260 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.797506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797484 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.797703 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797686 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.797879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797864 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.797976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797964 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.798070 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.798059 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9j2c\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.798188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.798176 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.798348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.798334 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.798480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.798467 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.798648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.797335 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 00:42:17.899929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.899890 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.899929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.899930 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.899960 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzkp\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900080 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9j2c\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900117 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900159 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900247 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900455 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900409 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900496 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900524 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900554 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900682 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.900873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900813 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.900873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900820 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.901032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900879 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.901032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.900931 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:17.901032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.901009 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.901142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.901042 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.901233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.901191 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.902962 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.902940 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.903382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.903364 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.921907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.921879 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:17.922111 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:17.922094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9j2c\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002567 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002715 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzkp\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002755 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002796 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002835 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002884 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002911 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.002979 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.003007 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.003612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.003685 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.003812 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.004159 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.004807 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.006779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.006333 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.007161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.007140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.026995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.026963 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.027142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.027015 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzkp\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.039938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.039895 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:18.105343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.105303 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:18.395136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.394975 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:42:18.398397 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:18.398367 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd667b8_103b_44e9_b8b7_9f633ff7c4f0.slice/crio-d4e3a7a15dc6e345356703e67e93a6218ad11aacd744e0d49c4a07d2a5956949 WatchSource:0}: Error finding container d4e3a7a15dc6e345356703e67e93a6218ad11aacd744e0d49c4a07d2a5956949: Status 404 returned error can't find the container with id d4e3a7a15dc6e345356703e67e93a6218ad11aacd744e0d49c4a07d2a5956949 Oct 31 00:42:18.401539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.401502 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.401633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.401563 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.401633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.401593 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.410539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.410516 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:42:18.415940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.415920 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.446395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.443761 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:42:18.452263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.450648 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:42:18.456664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.456642 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.468507 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:18.460545 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97d2c5b_f201_4aa7_bccb_e03b6cf994d1.slice/crio-7ccfa90a0ca49b913dd4e212ffec5477b175a1f4aefdc2f600bd4793b0e6e5c2 WatchSource:0}: Error finding container 7ccfa90a0ca49b913dd4e212ffec5477b175a1f4aefdc2f600bd4793b0e6e5c2: Status 404 returned error can't find the container with id 7ccfa90a0ca49b913dd4e212ffec5477b175a1f4aefdc2f600bd4793b0e6e5c2 Oct 31 00:42:18.468507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.461107 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 00:42:18.479423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.471929 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.479423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.471993 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.479423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.472034 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:18.479834 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.479769 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.510732 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.510861 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.510987 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511030 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511059 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511113 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511260 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5s8\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.511577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.511338 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612233 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612351 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612436 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9m7\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612478 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612509 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612533 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612583 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612614 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612641 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612672 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612710 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612742 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5s8\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612841 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612875 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612911 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.612943 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.613501 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.614547 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.615602 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.615900 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.627405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.616447 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.628505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.628045 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.632249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.631283 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.647380 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.646831 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.671096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.670961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5s8\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.714086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714047 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.714615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714294 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9m7\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.714692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714628 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.714692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.714808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714791 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715051 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.714826 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715222 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715258 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715291 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715476 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715622 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715701 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.715819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.715788 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.718358 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.718181 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.718604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.718580 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.757089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.744560 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.757089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.754144 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9m7\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.763769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.759906 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" event={"ID":"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0","Type":"ContainerStarted","Data":"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519"} Oct 31 00:42:18.763769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.759959 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" event={"ID":"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0","Type":"ContainerStarted","Data":"d4e3a7a15dc6e345356703e67e93a6218ad11aacd744e0d49c4a07d2a5956949"} Oct 31 00:42:18.763769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.762856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" event={"ID":"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1","Type":"ContainerStarted","Data":"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848"} Oct 31 00:42:18.763769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.762899 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" event={"ID":"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1","Type":"ContainerStarted","Data":"7ccfa90a0ca49b913dd4e212ffec5477b175a1f4aefdc2f600bd4793b0e6e5c2"} Oct 31 00:42:18.777466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.777395 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:18.800557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.800532 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:18.813039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.812984 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podStartSLOduration=1.812964832 podStartE2EDuration="1.812964832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:18.810946438 +0000 UTC m=+1407.260034080" watchObservedRunningTime="2026-10-31 00:42:18.812964832 +0000 UTC m=+1407.262052474" Oct 31 00:42:18.854533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:18.854464 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podStartSLOduration=1.854444242 podStartE2EDuration="1.854444242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:18.85305875 +0000 UTC m=+1407.302146389" watchObservedRunningTime="2026-10-31 00:42:18.854444242 +0000 UTC m=+1407.303531882" Oct 31 00:42:19.042497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.042411 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:19.043724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.043696 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:19.043863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.043753 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:19.106030 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.105999 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:19.107451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.107426 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:19.107536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.107481 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:19.188306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.188266 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:42:19.190319 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:19.190290 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bd0ace_323e_4af4_90bb_7e02686cc9a8.slice/crio-b201c967d2b2f30034f71f651536329eea9bb1b8fd79e807d4e8f33d47519f08 WatchSource:0}: Error finding container b201c967d2b2f30034f71f651536329eea9bb1b8fd79e807d4e8f33d47519f08: Status 404 returned error can't find the container with id b201c967d2b2f30034f71f651536329eea9bb1b8fd79e807d4e8f33d47519f08 Oct 31 00:42:19.192578 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:19.192550 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c8d202_c895_41f8_8204_b3f99251e1cf.slice/crio-6aa189b2e1ba2363208072dc98ddeabfa6dd280f278d0d3206f1b760441d7814 WatchSource:0}: Error finding container 6aa189b2e1ba2363208072dc98ddeabfa6dd280f278d0d3206f1b760441d7814: Status 404 returned error can't find the container with id 6aa189b2e1ba2363208072dc98ddeabfa6dd280f278d0d3206f1b760441d7814 Oct 31 00:42:19.193301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.193274 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.193370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.193348 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.193455 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.193394 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.193993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.193970 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:42:19.195105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.195080 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.195180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.195131 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.195180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.195163 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:19.913825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.912487 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" event={"ID":"09bd0ace-323e-4af4-90bb-7e02686cc9a8","Type":"ContainerStarted","Data":"d35c1f33da86b8240ae7c7547c850e7b31ae3af38f82327d3b96fc6c8270930b"} Oct 31 00:42:19.913825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.912560 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" event={"ID":"09bd0ace-323e-4af4-90bb-7e02686cc9a8","Type":"ContainerStarted","Data":"b201c967d2b2f30034f71f651536329eea9bb1b8fd79e807d4e8f33d47519f08"} Oct 31 00:42:19.917330 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.917302 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" event={"ID":"d5c8d202-c895-41f8-8204-b3f99251e1cf","Type":"ContainerStarted","Data":"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd"} Oct 31 00:42:19.949225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:19.949164 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" event={"ID":"d5c8d202-c895-41f8-8204-b3f99251e1cf","Type":"ContainerStarted","Data":"6aa189b2e1ba2363208072dc98ddeabfa6dd280f278d0d3206f1b760441d7814"} Oct 31 00:42:20.089321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.079952 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podStartSLOduration=2.079929823 podStartE2EDuration="2.079929823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:19.985555872 +0000 UTC m=+1408.434643512" watchObservedRunningTime="2026-10-31 00:42:20.079929823 +0000 UTC m=+1408.529017463" Oct 31 00:42:20.089321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.081529 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podStartSLOduration=2.081510857 podStartE2EDuration="2.081510857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:20.062909313 +0000 UTC m=+1408.511996953" watchObservedRunningTime="2026-10-31 00:42:20.081510857 +0000 UTC m=+1408.530598497" Oct 31 00:42:20.099229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.066196 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.099229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.096347 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:20.107682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.107597 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.107987 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.107952 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:20.788232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.787185 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:20.788633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.788604 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.788837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.788812 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:20.803513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.803480 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:20.806753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.805368 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.806977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:20.806954 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:21.042027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.041937 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.042627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.042587 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:21.128229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.112435 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.128229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.112504 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:21.640232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.634239 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:42:21.730229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.727293 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:21.773751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.773709 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:42:21.780817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.780784 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.781062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.781040 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:21.794229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.790280 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:21.804582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.804550 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.804809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.804787 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:21.842305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.840709 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:42:21.984290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:21.983309 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.040701 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.040662 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.040953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.040927 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:22.108245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.108184 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.108853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.108801 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:22.295410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.295302 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.362413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.362358 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.636706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.636659 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.637436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.637414 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:42:22.665331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.665288 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.715748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.702187 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.736560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.736507 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.737331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.737299 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:42:22.892362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.892250 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.893525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.892697 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:42:22.893525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.893084 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:42:22.893525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.893405 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:42:22.894570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.893851 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.894570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.894225 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:42:22.894570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.894306 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.894570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.894343 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:22.938284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.895094 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.938284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.895130 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:23.071582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.994838 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:42:23.071582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:22.995330 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:42:23.071582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.048409 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:23.071582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.048460 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:23.079726 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.078527 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:42:23.113832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.113796 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:23.114870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.113851 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:23.305294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.298549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.310490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.310448 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:42:23.364622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.364271 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.364622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.364478 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:42:23.698232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.645526 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:42:23.698232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:23.684470 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:42:24.031286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.027690 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.031286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.031095 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 31 00:42:24.061230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.031733 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.186526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.038608 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.190332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.190285 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:24.206229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.090441 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.274396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.264297 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:24.369225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.223394 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.369225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.251840 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:42:24.385427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.385382 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.387720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.133662 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.387915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.387890 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:24.388087 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.301816 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:42:24.390223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.325272 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.390377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.390356 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:24.390540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.110973 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:42:24.471243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.423225 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 31 00:42:24.647653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.647292 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:42:24.649245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.649195 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:42:24.711292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.702280 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:42:24.711292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.710337 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:42:24.760604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.760556 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:42:24.787238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.785111 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.787238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.786274 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.787238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.786315 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:24.854241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.844839 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.854505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:24.854463 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:25.040282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.033719 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.061238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.057586 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.069325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.069284 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:25.110329 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.110293 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.110587 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.110549 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:25.125876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.125830 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.126398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.126369 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:42:25.381231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.381128 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.381821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.381783 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.386301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.386269 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.392316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.392284 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.402304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.402266 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:42:25.700290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.690294 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.710467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.710427 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.762481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.762434 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:42:25.787166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.787130 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.803357 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.803300 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:25.805860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.805827 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.827231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.808988 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.829653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.829620 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:25.829838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.829349 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:42:25.829986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:25.829974 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:42:26.061306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.061193 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.061573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.061543 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:26.062085 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.062048 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.107646 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.107610 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.107894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.107868 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:26.206725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.206679 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.207267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.206792 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.207729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.207709 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:42:26.453487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.453440 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.499236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.495232 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:42:26.499236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.495625 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:42:26.517243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.516543 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:42:26.531273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.523830 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:42:26.720284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.695614 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.721455 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.784630 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.811748 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.784708 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.784679 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.808638 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.812072 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:26.813334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:26.807424 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.281231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.280839 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.319614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.310630 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.319614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.310693 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:27.319614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.314880 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.319614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.314944 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:27.319614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.315550 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:42:27.369675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.369635 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.378944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.370239 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:42:27.466054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.465276 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.505650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.497471 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.522565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.522520 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.523117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.523087 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 31 00:42:27.524168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.524142 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:42:27.546844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.546431 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.759964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.759932 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:42:27.760951 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.760929 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:42:27.786763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.786628 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.786763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.786689 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:27.794418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.794382 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:42:27.889612 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.889559 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.892332 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:27.917379 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.029318 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.038743 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.044968 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:42:28.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.045037 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 00:42:28.100449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.100414 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:42:28.117518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.117486 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:42:28.117768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.117746 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 00:42:28.209902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.194325 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:42:28.214932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.212101 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:42:28.225835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.225809 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:42:28.254609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.254585 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:42:28.267238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.265171 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:42:28.285161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.285135 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:42:28.285826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.285794 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:42:28.781014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.780982 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:28.781586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.781546 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:42:28.877713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.877601 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:28.877713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:28.877654 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:42:29.046918 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.046800 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:29.047704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.047681 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:29.048511 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.048491 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:42:29.110259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.110232 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:29.110608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.110583 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:29.111497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.111478 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 00:42:29.900476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.900434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:29.902086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.901970 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:29.904150 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.904127 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:42:29.967562 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.967531 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:29.968146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.968117 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:29.969479 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:29.969458 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:42:31.950412 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:42:31.950377 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:42:32.451261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.451224 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 00:42:32.456953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.456927 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.490242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.490197 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 00:42:32.536936 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.536897 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537064 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537095 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537163 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xh9m\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537231 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537310 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537338 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.537477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.537428 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638479 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638448 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638504 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638552 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638576 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638603 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xh9m\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638631 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.638666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638657 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638770 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638911 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.638981 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.639369 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.639490 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.639782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.639758 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.641217 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.641188 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.641487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.641464 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.661814 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.661784 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xh9m\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.662064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.662043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.772374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.772278 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:32.999327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:32.999271 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:42:33.011540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.011509 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.039750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.039672 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:42:33.127480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.127445 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 00:42:33.131735 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:33.131702 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2ed75c_5330_4a56_8ed6_f359274dee68.slice/crio-b523a96f116ec0dbda0bf0398ca80767d0a85b1ceee561bca94a9bdab77afac1 WatchSource:0}: Error finding container b523a96f116ec0dbda0bf0398ca80767d0a85b1ceee561bca94a9bdab77afac1: Status 404 returned error can't find the container with id b523a96f116ec0dbda0bf0398ca80767d0a85b1ceee561bca94a9bdab77afac1 Oct 31 00:42:33.134986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.134949 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.135098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.135031 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.135098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.135075 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.149975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.149947 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150026 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150116 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150092 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150122 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150311 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150380 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150362 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150389 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150480 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.150540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.150524 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252254 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252306 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252447 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252472 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252515 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252546 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252677 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252702 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.252881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.252776 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.253095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.253071 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.253404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.253375 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.253604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.253392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.253819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.253795 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.255853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.255829 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.255999 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.255978 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.274565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.274538 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.274819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.274733 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.355581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.355550 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:33.492991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.492961 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 00:42:33.500269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.498534 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.518641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.518250 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"distinct-2-w3-daaf26-kserve-self-signed-certs\"" Oct 31 00:42:33.525609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.525581 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.658933 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgxh\" (UniqueName: \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.659023 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.659047 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.659064 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.659087 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.659282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.659119 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.736230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.733622 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:42:33.762941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.762903 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgxh\" (UniqueName: \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.763367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.763340 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.763502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.763488 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.763629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.763615 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.763745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.763731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.763884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.763869 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.764565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.764539 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.765695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.765673 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.770286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.766081 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.785536 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.772761 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.791717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.788804 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.803254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.792193 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.803254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.792301 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.803254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.792344 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:42:33.816238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.813826 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgxh\" (UniqueName: \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\") pod \"distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:33.817257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:33.816957 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:34.187420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.186243 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" event={"ID":"8a2ed75c-5330-4a56-8ed6-f359274dee68","Type":"ContainerStarted","Data":"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4"} Oct 31 00:42:34.187420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.186311 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" event={"ID":"8a2ed75c-5330-4a56-8ed6-f359274dee68","Type":"ContainerStarted","Data":"b523a96f116ec0dbda0bf0398ca80767d0a85b1ceee561bca94a9bdab77afac1"} Oct 31 00:42:34.189841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.189359 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" event={"ID":"907a0c69-256f-40d9-b3d7-ea093bece237","Type":"ContainerStarted","Data":"a4285f9f76a02ae66f5182ea42cce23e0be7e74b00b13a754d9fce5948bf5d28"} Oct 31 00:42:34.189841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.189410 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" event={"ID":"907a0c69-256f-40d9-b3d7-ea093bece237","Type":"ContainerStarted","Data":"2b72cb04f055e418569001db34a8b324e64abbae825067b07a3154e639e5502e"} Oct 31 00:42:34.260232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.245324 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podStartSLOduration=2.245300522 podStartE2EDuration="2.245300522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:34.235908478 +0000 UTC m=+1422.684996119" watchObservedRunningTime="2026-10-31 00:42:34.245300522 +0000 UTC m=+1422.694388165" Oct 31 00:42:34.292073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.292023 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podStartSLOduration=2.292008757 podStartE2EDuration="2.292008757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:34.288631941 +0000 UTC m=+1422.737719594" watchObservedRunningTime="2026-10-31 00:42:34.292008757 +0000 UTC m=+1422.741096463" Oct 31 00:42:34.298720 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:34.298688 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb915eb_5c6f_4f70_99a8_7a7bdac856a6.slice/crio-dcbb5f8341d9d70883aa592d6300fcadc0d665974adb6403905aef6166653ee9 WatchSource:0}: Error finding container dcbb5f8341d9d70883aa592d6300fcadc0d665974adb6403905aef6166653ee9: Status 404 returned error can't find the container with id dcbb5f8341d9d70883aa592d6300fcadc0d665974adb6403905aef6166653ee9 Oct 31 00:42:34.299414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.299391 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 00:42:34.360258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.355891 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:34.777525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.777475 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:34.778069 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.777884 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:34.778069 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:34.777930 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:35.197959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.197851 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" event={"ID":"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6","Type":"ContainerStarted","Data":"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf"} Oct 31 00:42:35.197959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.197893 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" event={"ID":"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6","Type":"ContainerStarted","Data":"dcbb5f8341d9d70883aa592d6300fcadc0d665974adb6403905aef6166653ee9"} Oct 31 00:42:35.239229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.238775 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podStartSLOduration=2.238755091 podStartE2EDuration="2.238755091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:35.236597553 +0000 UTC m=+1423.685685196" watchObservedRunningTime="2026-10-31 00:42:35.238755091 +0000 UTC m=+1423.687842731" Oct 31 00:42:35.367375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.367334 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:35.367661 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.367628 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:35.777248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.777057 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:35.777248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.777119 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:35.840232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:35.839297 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:36.358230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:36.357553 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:36.358230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:36.357610 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:36.774286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:36.773669 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:36.774286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:36.773731 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:37.361233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:37.356530 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:37.361233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:37.356591 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:37.780282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:37.779557 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:37.780282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:37.779617 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:38.358905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:38.357555 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:38.358905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:38.357621 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:38.775697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:38.775612 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:38.776280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:38.776249 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:39.360653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:39.360612 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:39.360952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:39.360920 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:39.772645 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:39.772553 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:42:39.772645 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:39.772601 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 00:42:40.356488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:40.356448 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:40.356987 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:40.356542 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 00:42:40.866112 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:40.866071 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:40.867134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:40.867099 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:40.868018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:40.867991 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 00:42:41.360936 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:41.360904 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:41.361449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:41.361374 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:41.362163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:41.362144 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:42:43.817948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:43.817917 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:43.830970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:43.830944 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:44.331155 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:44.331122 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:42:47.895344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.895310 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt"] Oct 31 00:42:47.899623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.899607 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:47.917392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.917369 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt"] Oct 31 00:42:47.922783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.922758 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:47.922885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.922813 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:42:47.935338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.935309 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:47.935462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.935354 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxt6w\" (UniqueName: \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:47.935632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:47.935615 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.036754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.036720 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.036891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.036854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.036932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.036891 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxt6w\" (UniqueName: \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.037514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.037474 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.039473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.039449 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.059365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.059345 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxt6w\" (UniqueName: \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\") pod \"maas-api-revoke-sub-851dae1f-k48tt\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.214229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.214142 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:48.525883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:48.525853 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt"] Oct 31 00:42:48.527546 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:42:48.527502 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9df0724_7a93_42a6_94f4_451e184f5ff3.slice/crio-491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d WatchSource:0}: Error finding container 491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d: Status 404 returned error can't find the container with id 491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d Oct 31 00:42:49.361182 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:49.361144 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" event={"ID":"c9df0724-7a93-42a6-94f4-451e184f5ff3","Type":"ContainerStarted","Data":"1b0dd36d94d6b7e4322f5d1d6f3e1289468f8319c1a7c8e72406b3b686abb69a"} Oct 31 00:42:49.361712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:49.361230 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" event={"ID":"c9df0724-7a93-42a6-94f4-451e184f5ff3","Type":"ContainerStarted","Data":"491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d"} Oct 31 00:42:49.404985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:49.404925 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" podStartSLOduration=2.404906281 podStartE2EDuration="2.404906281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:49.400229099 +0000 UTC m=+1437.849316741" watchObservedRunningTime="2026-10-31 00:42:49.404906281 +0000 UTC m=+1437.853993923" Oct 31 00:42:50.388016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:50.387984 5011 generic.go:356] "Generic (PLEG): container finished" podID="c9df0724-7a93-42a6-94f4-451e184f5ff3" containerID="1b0dd36d94d6b7e4322f5d1d6f3e1289468f8319c1a7c8e72406b3b686abb69a" exitCode=0 Oct 31 00:42:50.388542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:50.388109 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" event={"ID":"c9df0724-7a93-42a6-94f4-451e184f5ff3","Type":"ContainerDied","Data":"1b0dd36d94d6b7e4322f5d1d6f3e1289468f8319c1a7c8e72406b3b686abb69a"} Oct 31 00:42:51.566580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.566552 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:51.592285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.592259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\") pod \"c9df0724-7a93-42a6-94f4-451e184f5ff3\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " Oct 31 00:42:51.592382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.592324 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\") pod \"c9df0724-7a93-42a6-94f4-451e184f5ff3\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " Oct 31 00:42:51.592434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.592405 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\" (UniqueName: \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\") pod \"c9df0724-7a93-42a6-94f4-451e184f5ff3\" (UID: \"c9df0724-7a93-42a6-94f4-451e184f5ff3\") " Oct 31 00:42:51.592739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.592712 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca" pod "c9df0724-7a93-42a6-94f4-451e184f5ff3" (UID: "c9df0724-7a93-42a6-94f4-451e184f5ff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:51.593047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.593027 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:42:51.594438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.594408 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca" pod "c9df0724-7a93-42a6-94f4-451e184f5ff3" (UID: "c9df0724-7a93-42a6-94f4-451e184f5ff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:51.594580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.594554 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w" pod "c9df0724-7a93-42a6-94f4-451e184f5ff3" (UID: "c9df0724-7a93-42a6-94f4-451e184f5ff3"). InnerVolumeSpecName "kube-api-access-wxt6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:51.694189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.694087 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9df0724-7a93-42a6-94f4-451e184f5ff3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:42:51.694189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:51.694122 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxt6w\" (UniqueName: \"kubernetes.io/projected/c9df0724-7a93-42a6-94f4-451e184f5ff3-kube-api-access-wxt6w\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:42:52.405294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:52.405264 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" Oct 31 00:42:52.405462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:52.405260 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt" event={"ID":"c9df0724-7a93-42a6-94f4-451e184f5ff3","Type":"ContainerDied","Data":"491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d"} Oct 31 00:42:52.405462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:52.405367 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491a5eaa112d04c4ac992cfa07d3b6c341610b0ed677035af460c492df9fc03d" Oct 31 00:42:59.924313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:42:59.924278 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:43:02.366896 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:02.366856 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:43:02.517113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:02.517069 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.039557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.038301 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.039915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.039864 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:43:03.137382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.137337 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.332452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.332405 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:43:03.333446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.333412 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.437323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.426606 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.442239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.438935 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.501232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.475993 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.731289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.727682 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:43:03.830353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.830302 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.979771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.976480 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 00:43:03.979771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.977448 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c9df0724-7a93-42a6-94f4-451e184f5ff3" containerName="revoke-keys" Oct 31 00:43:03.979771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.977467 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9df0724-7a93-42a6-94f4-451e184f5ff3" containerName="revoke-keys" Oct 31 00:43:03.979771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:03.977685 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9df0724-7a93-42a6-94f4-451e184f5ff3" containerName="revoke-keys" Oct 31 00:43:04.009288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.004487 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.072353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.072301 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.072940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.072915 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:43:04.073745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.073632 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:43:04.073931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.073918 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:43:04.097540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.097303 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 00:43:04.180231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.137286 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:43:04.288287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.184934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.295851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.295808 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.335578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.335541 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.354978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355035 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355166 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355221 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355265 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355386 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355461 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.369227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.355593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktt6w\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.432291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.428518 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:43:04.440616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.440579 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.486416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.486380 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.486736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.486696 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.487001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.486980 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt6w\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.487334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.487312 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.487493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.487476 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.487720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.487702 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.488308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.488287 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.488478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.488459 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.488688 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.488667 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.492305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.492273 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.505489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.505464 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.536845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.536706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.543049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.542976 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.544303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.544186 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.579020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.546511 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.579020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.578452 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.620231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.604607 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:43:04.639371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.639341 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:43:04.639693 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.639677 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:43:04.640098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.640080 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:43:04.703606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.703570 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:43:04.704576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.704557 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:43:04.705445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.705427 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:43:04.705597 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.606441 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.706429 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.698523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.706612 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.706593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.706783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.706753 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.706971 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.707122 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.707260 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltr5f\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.707430 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.707658 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.707888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.707873 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.708107 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.652942 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktt6w\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.708375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.635542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.713295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.712986 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:04.783445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.783386 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:43:04.785628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.785598 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:43:04.813148 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.812896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.813530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.813507 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.813714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.813698 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.813841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.813825 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.813999 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.813967 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.814249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.814230 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.814396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.814377 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.814524 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.814506 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltr5f\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.814698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.814681 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:04.848439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.816921 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:43:05.062283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:04.981026 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 00:43:05.074233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.037504 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.086231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.053497 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.096138 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.104964 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.105228 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.116533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.247904 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.282331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.282386 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.289886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltr5f\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.357335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.321142 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:05.359861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.358147 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:43:05.483505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.474434 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:05.559609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.555652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.623246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.620913 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:43:05.659790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.658724 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:43:05.722594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.722503 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:05.834164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.831955 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:43:05.834164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.832023 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:43:05.937817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:05.937727 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:43:06.093666 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:06.093622 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53214e19_073f_4c12_928b_51799031a596.slice/crio-ee9a60bbf99665ffbbddf087ebc2a96140fb7896797a6cab547edfe847182b27 WatchSource:0}: Error finding container ee9a60bbf99665ffbbddf087ebc2a96140fb7896797a6cab547edfe847182b27: Status 404 returned error can't find the container with id ee9a60bbf99665ffbbddf087ebc2a96140fb7896797a6cab547edfe847182b27 Oct 31 00:43:06.097989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.097864 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 00:43:06.102022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.101981 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.102304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.102278 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.102469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.102451 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.308129 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.308084 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:43:06.315347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.315312 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.321237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.319328 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.321237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.319387 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:06.507993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.507946 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" event={"ID":"16a9b271-9aa0-43ec-85b9-956294374f3c","Type":"ContainerStarted","Data":"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88"} Oct 31 00:43:06.508965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.508859 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh"] Oct 31 00:43:06.525321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.519472 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" event={"ID":"16a9b271-9aa0-43ec-85b9-956294374f3c","Type":"ContainerStarted","Data":"4cb84f6ed77ba320b2904e12f08b5a29b62e76c59bbd301185e22784b5a4eb73"} Oct 31 00:43:06.525321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.519866 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.528634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.527592 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" event={"ID":"53214e19-073f-4c12-928b-51799031a596","Type":"ContainerStarted","Data":"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207"} Oct 31 00:43:06.528634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.527656 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" event={"ID":"53214e19-073f-4c12-928b-51799031a596","Type":"ContainerStarted","Data":"ee9a60bbf99665ffbbddf087ebc2a96140fb7896797a6cab547edfe847182b27"} Oct 31 00:43:06.534238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.532077 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh"] Oct 31 00:43:06.561347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.560740 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podStartSLOduration=3.560719903 podStartE2EDuration="3.560719903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:06.557007327 +0000 UTC m=+1455.006094966" watchObservedRunningTime="2026-10-31 00:43:06.560719903 +0000 UTC m=+1455.009807544" Oct 31 00:43:06.570823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.570068 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:06.570823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.570234 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:06.599278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.599248 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.599832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.599805 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.600029 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.600012 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnblt\" (UniqueName: \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.648253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.648176 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podStartSLOduration=3.6481543260000002 podStartE2EDuration="3.648154326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:06.639702455 +0000 UTC m=+1455.088790094" watchObservedRunningTime="2026-10-31 00:43:06.648154326 +0000 UTC m=+1455.097241968" Oct 31 00:43:06.701841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.701769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.702051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.702038 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnblt\" (UniqueName: \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.702174 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.702164 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.703425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.703407 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.704983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.704967 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.722934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.722899 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:06.723331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.723307 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:06.733925 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.733895 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnblt\" (UniqueName: \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\") pod \"maas-api-revoke-sub-c7a56df4-j9bfh\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:06.864261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:06.857535 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:07.176857 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.176828 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh"] Oct 31 00:43:07.178316 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:07.178290 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62b60de_2fa3_4352_bde4_2940889ef475.slice/crio-0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57 WatchSource:0}: Error finding container 0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57: Status 404 returned error can't find the container with id 0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57 Oct 31 00:43:07.604541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.593537 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" event={"ID":"b62b60de-2fa3-4352-bde4-2940889ef475","Type":"ContainerStarted","Data":"0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57"} Oct 31 00:43:07.765287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.761872 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:07.765287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.761931 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:07.778250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.776577 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:07.778250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:07.776638 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:08.651291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.636479 5011 generic.go:356] "Generic (PLEG): container finished" podID="b62b60de-2fa3-4352-bde4-2940889ef475" containerID="975ac4d14bba38543bf9107acd6ba1645144beb9da0016656088ca36b53865af" exitCode=0 Oct 31 00:43:08.651291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.637870 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" event={"ID":"b62b60de-2fa3-4352-bde4-2940889ef475","Type":"ContainerDied","Data":"975ac4d14bba38543bf9107acd6ba1645144beb9da0016656088ca36b53865af"} Oct 31 00:43:08.809311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.807858 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.809311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.808120 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:08.809311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.808791 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.809311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:08.808857 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:09.730232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:09.728998 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.730232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:09.729073 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:09.743229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:09.730892 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.743229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:09.730946 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:10.074447 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.074401 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:10.195973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.195940 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\") pod \"b62b60de-2fa3-4352-bde4-2940889ef475\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " Oct 31 00:43:10.196254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.196229 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\") pod \"b62b60de-2fa3-4352-bde4-2940889ef475\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " Oct 31 00:43:10.196337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.196313 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\" (UniqueName: \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\") pod \"b62b60de-2fa3-4352-bde4-2940889ef475\" (UID: \"b62b60de-2fa3-4352-bde4-2940889ef475\") " Oct 31 00:43:10.197010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.196982 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca" pod "b62b60de-2fa3-4352-bde4-2940889ef475" (UID: "b62b60de-2fa3-4352-bde4-2940889ef475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:10.198443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.198423 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca" pod "b62b60de-2fa3-4352-bde4-2940889ef475" (UID: "b62b60de-2fa3-4352-bde4-2940889ef475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:10.198627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.198594 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt" pod "b62b60de-2fa3-4352-bde4-2940889ef475" (UID: "b62b60de-2fa3-4352-bde4-2940889ef475"). InnerVolumeSpecName "kube-api-access-cnblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:10.299592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.298065 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:10.299592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.298100 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnblt\" (UniqueName: \"kubernetes.io/projected/b62b60de-2fa3-4352-bde4-2940889ef475-kube-api-access-cnblt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:10.299592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.298115 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62b60de-2fa3-4352-bde4-2940889ef475-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:10.502862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.502686 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd"] Oct 31 00:43:10.503583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.503515 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b62b60de-2fa3-4352-bde4-2940889ef475" containerName="revoke-keys" Oct 31 00:43:10.503583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.503535 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62b60de-2fa3-4352-bde4-2940889ef475" containerName="revoke-keys" Oct 31 00:43:10.503768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.503749 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62b60de-2fa3-4352-bde4-2940889ef475" containerName="revoke-keys" Oct 31 00:43:10.508384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.508360 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.550722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.517023 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd"] Oct 31 00:43:10.613388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.611431 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.613388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.611772 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn29r\" (UniqueName: \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.613388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.611816 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.712423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.712385 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" event={"ID":"b62b60de-2fa3-4352-bde4-2940889ef475","Type":"ContainerDied","Data":"0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57"} Oct 31 00:43:10.712590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.712447 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0112f202e700f86024029cc80f66e2f7ecb2134ce17d7eddb6db429b3dcdee57" Oct 31 00:43:10.712590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.712573 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh" Oct 31 00:43:10.713053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.713024 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn29r\" (UniqueName: \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.713102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.713086 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.713179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.713163 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.714092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.714070 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.716627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.716602 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.735270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.727604 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.735270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.727660 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:10.735270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.728258 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.735270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.728295 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:10.807017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.805189 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn29r\" (UniqueName: \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\") pod \"maas-api-revoke-sub-47038305-8cxqd\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:10.912835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:10.912798 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:11.380290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.379943 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd"] Oct 31 00:43:11.749938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.748580 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.749938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.748644 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:11.762238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.758331 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.762238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.758391 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:11.782271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:11.774909 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" event={"ID":"f0daf7b5-680e-4dd4-b240-eee98313c19b","Type":"ContainerStarted","Data":"062cc4061602eee6a9b16fff48b002633f6bf01e2dc03016658ec4c2cf86f8ae"} Oct 31 00:43:12.722608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.722573 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.742111 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.742069 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:43:12.742747 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.724745 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.746320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.746287 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:12.787074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.787041 5011 generic.go:356] "Generic (PLEG): container finished" podID="f0daf7b5-680e-4dd4-b240-eee98313c19b" containerID="95cc68dab97ede431881ce707dbacddb6588e391794a020f7f3b8efd35e5da0b" exitCode=0 Oct 31 00:43:12.787600 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:12.787191 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" event={"ID":"f0daf7b5-680e-4dd4-b240-eee98313c19b","Type":"ContainerDied","Data":"95cc68dab97ede431881ce707dbacddb6588e391794a020f7f3b8efd35e5da0b"} Oct 31 00:43:13.723625 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:13.723572 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:43:13.723811 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:13.723645 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:43:14.128179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.128137 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:14.236462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.236419 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\") pod \"f0daf7b5-680e-4dd4-b240-eee98313c19b\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " Oct 31 00:43:14.236662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.236546 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\" (UniqueName: \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\") pod \"f0daf7b5-680e-4dd4-b240-eee98313c19b\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " Oct 31 00:43:14.236817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.236791 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\") pod \"f0daf7b5-680e-4dd4-b240-eee98313c19b\" (UID: \"f0daf7b5-680e-4dd4-b240-eee98313c19b\") " Oct 31 00:43:14.237696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.237674 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca" pod "f0daf7b5-680e-4dd4-b240-eee98313c19b" (UID: "f0daf7b5-680e-4dd4-b240-eee98313c19b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:14.242186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.242157 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:14.244357 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.244332 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:14.245458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.245434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 00:43:14.247581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.246329 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca" pod "f0daf7b5-680e-4dd4-b240-eee98313c19b" (UID: "f0daf7b5-680e-4dd4-b240-eee98313c19b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:14.248675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.248634 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r" pod "f0daf7b5-680e-4dd4-b240-eee98313c19b" (UID: "f0daf7b5-680e-4dd4-b240-eee98313c19b"). InnerVolumeSpecName "kube-api-access-qn29r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:14.339849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.339773 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:14.340676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.340655 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn29r\" (UniqueName: \"kubernetes.io/projected/f0daf7b5-680e-4dd4-b240-eee98313c19b-kube-api-access-qn29r\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:14.340835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.340819 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0daf7b5-680e-4dd4-b240-eee98313c19b-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:14.728186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.728108 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:14.728639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.728623 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:14.729309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.729291 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:43:14.810738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.810705 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" Oct 31 00:43:14.811668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.810921 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd" event={"ID":"f0daf7b5-680e-4dd4-b240-eee98313c19b","Type":"ContainerDied","Data":"062cc4061602eee6a9b16fff48b002633f6bf01e2dc03016658ec4c2cf86f8ae"} Oct 31 00:43:14.811818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.811688 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062cc4061602eee6a9b16fff48b002633f6bf01e2dc03016658ec4c2cf86f8ae" Oct 31 00:43:14.920092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.920053 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm"] Oct 31 00:43:14.920763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.920741 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f0daf7b5-680e-4dd4-b240-eee98313c19b" containerName="revoke-keys" Oct 31 00:43:14.920763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.920763 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0daf7b5-680e-4dd4-b240-eee98313c19b" containerName="revoke-keys" Oct 31 00:43:14.920957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.920943 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0daf7b5-680e-4dd4-b240-eee98313c19b" containerName="revoke-keys" Oct 31 00:43:14.924886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.924863 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:14.957555 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.957526 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:14.957738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.957583 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvz9f\" (UniqueName: \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:14.957810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.957730 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:14.963838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.963786 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:14.964121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.964099 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:14.968973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:14.968943 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm"] Oct 31 00:43:15.059573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.059493 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.059573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.059545 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvz9f\" (UniqueName: \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.059755 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.059591 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.060319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.060289 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.062234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.062195 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.081656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.081630 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvz9f\" (UniqueName: \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\") pod \"maas-api-revoke-sub-3f5a410a-mhgdm\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.246239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.246197 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:15.559736 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:15.559699 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b50d89_eb62_46c0_8627_0cc9d7f001e4.slice/crio-b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b WatchSource:0}: Error finding container b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b: Status 404 returned error can't find the container with id b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b Oct 31 00:43:15.561028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.561000 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm"] Oct 31 00:43:15.821675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:15.821638 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" event={"ID":"43b50d89-eb62-46c0-8627-0cc9d7f001e4","Type":"ContainerStarted","Data":"b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b"} Oct 31 00:43:16.832695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:16.832660 5011 generic.go:356] "Generic (PLEG): container finished" podID="43b50d89-eb62-46c0-8627-0cc9d7f001e4" containerID="8246145c2768d6a5da7bcdcf440a2c21bce7557cd6c1bcb273fb6d246386f374" exitCode=0 Oct 31 00:43:16.833128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:16.832749 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" event={"ID":"43b50d89-eb62-46c0-8627-0cc9d7f001e4","Type":"ContainerDied","Data":"8246145c2768d6a5da7bcdcf440a2c21bce7557cd6c1bcb273fb6d246386f374"} Oct 31 00:43:17.940581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:17.940544 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:43:17.945170 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:17.945140 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:17.972822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:17.972791 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:43:18.008126 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008080 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008146 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008183 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008249 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008414 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008455 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008589 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008654 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftkp\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.008704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.008683 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.051983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.051958 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:18.109307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.109199 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\") pod \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " Oct 31 00:43:18.109307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.109290 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\") pod \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " Oct 31 00:43:18.109540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.109524 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\" (UniqueName: \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\") pod \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\" (UID: \"43b50d89-eb62-46c0-8627-0cc9d7f001e4\") " Oct 31 00:43:18.109653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.109601 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca" pod "43b50d89-eb62-46c0-8627-0cc9d7f001e4" (UID: "43b50d89-eb62-46c0-8627-0cc9d7f001e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:18.110043 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110014 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110060 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110092 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110143 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110201 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110310 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110454 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110473 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110523 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftkp\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110555 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110648 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:18.110774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110746 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.110973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.110950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.111246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.111227 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.111867 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.111811 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.111981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.111956 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca" pod "43b50d89-eb62-46c0-8627-0cc9d7f001e4" (UID: "43b50d89-eb62-46c0-8627-0cc9d7f001e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:18.113982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.113949 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f" pod "43b50d89-eb62-46c0-8627-0cc9d7f001e4" (UID: "43b50d89-eb62-46c0-8627-0cc9d7f001e4"). InnerVolumeSpecName "kube-api-access-rvz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:18.115607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.115585 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.115948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.115922 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.137072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.137032 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftkp\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.139074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.139049 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.211882 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.211845 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b50d89-eb62-46c0-8627-0cc9d7f001e4-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:18.211882 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.211876 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvz9f\" (UniqueName: \"kubernetes.io/projected/43b50d89-eb62-46c0-8627-0cc9d7f001e4-kube-api-access-rvz9f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:18.263601 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.263564 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:18.597429 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.597398 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:43:18.597555 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:18.597406 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b67d25_03a6_40c6_948b_f9d3821a63b8.slice/crio-6e29ffe8e64d1729d2f5addb303737735464086eec5c339d13322adfa75c9ebd WatchSource:0}: Error finding container 6e29ffe8e64d1729d2f5addb303737735464086eec5c339d13322adfa75c9ebd: Status 404 returned error can't find the container with id 6e29ffe8e64d1729d2f5addb303737735464086eec5c339d13322adfa75c9ebd Oct 31 00:43:18.600007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.599974 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:18.600105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.600037 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:18.600105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.600068 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:18.852739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.852656 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" Oct 31 00:43:18.852910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.852754 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm" event={"ID":"43b50d89-eb62-46c0-8627-0cc9d7f001e4","Type":"ContainerDied","Data":"b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b"} Oct 31 00:43:18.852910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.852811 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04784419e560eda0d3ac625be097bd05f25d61009449bb7f9c1f36444c1061b" Oct 31 00:43:18.855257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.855193 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" event={"ID":"d1b67d25-03a6-40c6-948b-f9d3821a63b8","Type":"ContainerStarted","Data":"a43416f8e50e27bf26c3155ef298a98536686b5972e53f731eda44eb1a0f336c"} Oct 31 00:43:18.855419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.855313 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" event={"ID":"d1b67d25-03a6-40c6-948b-f9d3821a63b8","Type":"ContainerStarted","Data":"6e29ffe8e64d1729d2f5addb303737735464086eec5c339d13322adfa75c9ebd"} Oct 31 00:43:18.905800 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:18.905733 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podStartSLOduration=1.905711657 podStartE2EDuration="1.905711657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:18.896791219 +0000 UTC m=+1467.345878860" watchObservedRunningTime="2026-10-31 00:43:18.905711657 +0000 UTC m=+1467.354799296" Oct 31 00:43:19.264793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.264707 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:19.265941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.265923 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:19.266001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.265964 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:19.278100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.278064 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn"] Oct 31 00:43:19.279003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.278986 5011 cpu_manager.go:414] "containerMap: removing container" podUID="43b50d89-eb62-46c0-8627-0cc9d7f001e4" containerName="revoke-keys" Oct 31 00:43:19.279060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.279007 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b50d89-eb62-46c0-8627-0cc9d7f001e4" containerName="revoke-keys" Oct 31 00:43:19.279160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.279148 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b50d89-eb62-46c0-8627-0cc9d7f001e4" containerName="revoke-keys" Oct 31 00:43:19.283505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.283488 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.299763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.299737 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:19.300012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.299990 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:19.301099 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.301078 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn"] Oct 31 00:43:19.443696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.443650 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.444165 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.444137 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.444445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.444425 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr69\" (UniqueName: \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.549315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.549169 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr69\" (UniqueName: \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.549315 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.549312 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.549587 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.549517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.550431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.550402 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.553258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.552789 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.573590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.573559 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr69\" (UniqueName: \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\") pod \"maas-api-revoke-sub-ad8df849-n9knn\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.598574 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.598544 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:19.949639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:19.949599 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn"] Oct 31 00:43:19.954453 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:19.954428 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907c6864_8500_4b31_9e5d_e6010e824b78.slice/crio-09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72 WatchSource:0}: Error finding container 09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72: Status 404 returned error can't find the container with id 09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72 Oct 31 00:43:20.264986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:20.264903 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:20.264986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:20.264951 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:20.878469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:20.878432 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" event={"ID":"907c6864-8500-4b31-9e5d-e6010e824b78","Type":"ContainerStarted","Data":"ce1be5d51d6e6c2401dc6919b3fbcd4bd57e4afd029d67144bd1a461d17cb827"} Oct 31 00:43:20.878734 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:20.878705 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" event={"ID":"907c6864-8500-4b31-9e5d-e6010e824b78","Type":"ContainerStarted","Data":"09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72"} Oct 31 00:43:20.948230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:20.946755 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" podStartSLOduration=1.946733054 podStartE2EDuration="1.946733054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:20.91488949 +0000 UTC m=+1469.363977131" watchObservedRunningTime="2026-10-31 00:43:20.946733054 +0000 UTC m=+1469.395820694" Oct 31 00:43:21.067753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.067439 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d"] Oct 31 00:43:21.071943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.071919 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.087825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.087801 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d"] Oct 31 00:43:21.177961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.177930 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.178106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.178026 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.178231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.178200 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crnb8\" (UniqueName: \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.264552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.264517 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:21.264734 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.264571 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:21.279745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.279718 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.280161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.279773 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.280161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.279891 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crnb8\" (UniqueName: \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.280537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.280516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.282200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.282174 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.301701 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.301675 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crnb8\" (UniqueName: \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\") pod \"maas-api-revoke-sub-e87ee625-9lp8d\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.475311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.475189 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:21.816970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.816831 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d"] Oct 31 00:43:21.817083 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:21.816978 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630e2837_390c_462b_9d0a_466237913999.slice/crio-6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78 WatchSource:0}: Error finding container 6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78: Status 404 returned error can't find the container with id 6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78 Oct 31 00:43:21.887456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.887424 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" event={"ID":"630e2837-390c-462b-9d0a-466237913999","Type":"ContainerStarted","Data":"6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78"} Oct 31 00:43:21.889837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.889811 5011 generic.go:356] "Generic (PLEG): container finished" podID="907c6864-8500-4b31-9e5d-e6010e824b78" containerID="ce1be5d51d6e6c2401dc6919b3fbcd4bd57e4afd029d67144bd1a461d17cb827" exitCode=0 Oct 31 00:43:21.889935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:21.889923 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" event={"ID":"907c6864-8500-4b31-9e5d-e6010e824b78","Type":"ContainerDied","Data":"ce1be5d51d6e6c2401dc6919b3fbcd4bd57e4afd029d67144bd1a461d17cb827"} Oct 31 00:43:22.264368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:22.264332 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:22.264545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:22.264383 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:22.900354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:22.900320 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" event={"ID":"630e2837-390c-462b-9d0a-466237913999","Type":"ContainerStarted","Data":"44988df288e1b4e8e5dcad017969fdf1637bc67e31bcc8a2937e018bd3c44b89"} Oct 31 00:43:22.946226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:22.937993 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" podStartSLOduration=1.937975867 podStartE2EDuration="1.937975867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:22.937608435 +0000 UTC m=+1471.386696075" watchObservedRunningTime="2026-10-31 00:43:22.937975867 +0000 UTC m=+1471.387063508" Oct 31 00:43:23.153738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.153665 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:23.264785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.264733 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:23.264958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.264803 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:23.313167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.313137 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\" (UniqueName: \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\") pod \"907c6864-8500-4b31-9e5d-e6010e824b78\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " Oct 31 00:43:23.313340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.313300 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\") pod \"907c6864-8500-4b31-9e5d-e6010e824b78\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " Oct 31 00:43:23.313340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.313330 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\") pod \"907c6864-8500-4b31-9e5d-e6010e824b78\" (UID: \"907c6864-8500-4b31-9e5d-e6010e824b78\") " Oct 31 00:43:23.313832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.313787 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca" pod "907c6864-8500-4b31-9e5d-e6010e824b78" (UID: "907c6864-8500-4b31-9e5d-e6010e824b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:23.315718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.315696 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca" pod "907c6864-8500-4b31-9e5d-e6010e824b78" (UID: "907c6864-8500-4b31-9e5d-e6010e824b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:23.315835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.315819 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69" pod "907c6864-8500-4b31-9e5d-e6010e824b78" (UID: "907c6864-8500-4b31-9e5d-e6010e824b78"). InnerVolumeSpecName "kube-api-access-scr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:23.415113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.415035 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr69\" (UniqueName: \"kubernetes.io/projected/907c6864-8500-4b31-9e5d-e6010e824b78-kube-api-access-scr69\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:23.415113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.415064 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:23.415113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.415076 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c6864-8500-4b31-9e5d-e6010e824b78-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:23.909683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.909648 5011 generic.go:356] "Generic (PLEG): container finished" podID="630e2837-390c-462b-9d0a-466237913999" containerID="44988df288e1b4e8e5dcad017969fdf1637bc67e31bcc8a2937e018bd3c44b89" exitCode=0 Oct 31 00:43:23.910042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.909736 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" event={"ID":"630e2837-390c-462b-9d0a-466237913999","Type":"ContainerDied","Data":"44988df288e1b4e8e5dcad017969fdf1637bc67e31bcc8a2937e018bd3c44b89"} Oct 31 00:43:23.911249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.911223 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" event={"ID":"907c6864-8500-4b31-9e5d-e6010e824b78","Type":"ContainerDied","Data":"09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72"} Oct 31 00:43:23.911348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.911265 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d0ef42008c153a727dbfad58b7ba9992b255536179df6d2c91a8b6bb10ff72" Oct 31 00:43:23.911426 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:23.911412 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn" Oct 31 00:43:24.264256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:24.264190 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:24.264433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:24.264286 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:25.100756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.100729 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:25.169399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.169355 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\" (UniqueName: \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\") pod \"630e2837-390c-462b-9d0a-466237913999\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " Oct 31 00:43:25.169566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.169534 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\") pod \"630e2837-390c-462b-9d0a-466237913999\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " Oct 31 00:43:25.169662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.169648 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\") pod \"630e2837-390c-462b-9d0a-466237913999\" (UID: \"630e2837-390c-462b-9d0a-466237913999\") " Oct 31 00:43:25.173631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.171326 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca" pod "630e2837-390c-462b-9d0a-466237913999" (UID: "630e2837-390c-462b-9d0a-466237913999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:25.190774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.190749 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca" pod "630e2837-390c-462b-9d0a-466237913999" (UID: "630e2837-390c-462b-9d0a-466237913999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:25.190859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.190795 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8" pod "630e2837-390c-462b-9d0a-466237913999" (UID: "630e2837-390c-462b-9d0a-466237913999"). InnerVolumeSpecName "kube-api-access-crnb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:25.263946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.263911 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:43:25.264104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.263961 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 00:43:25.271492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.271463 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e2837-390c-462b-9d0a-466237913999-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:25.271492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.271493 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e2837-390c-462b-9d0a-466237913999-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:25.271704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.271508 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crnb8\" (UniqueName: \"kubernetes.io/projected/630e2837-390c-462b-9d0a-466237913999-kube-api-access-crnb8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:25.322883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.322851 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww"] Oct 31 00:43:25.323804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.323783 5011 cpu_manager.go:414] "containerMap: removing container" podUID="630e2837-390c-462b-9d0a-466237913999" containerName="revoke-keys" Oct 31 00:43:25.323907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.323806 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630e2837-390c-462b-9d0a-466237913999" containerName="revoke-keys" Oct 31 00:43:25.323907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.323828 5011 cpu_manager.go:414] "containerMap: removing container" podUID="907c6864-8500-4b31-9e5d-e6010e824b78" containerName="revoke-keys" Oct 31 00:43:25.323907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.323836 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907c6864-8500-4b31-9e5d-e6010e824b78" containerName="revoke-keys" Oct 31 00:43:25.324065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.324024 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="630e2837-390c-462b-9d0a-466237913999" containerName="revoke-keys" Oct 31 00:43:25.324065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.324037 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="907c6864-8500-4b31-9e5d-e6010e824b78" containerName="revoke-keys" Oct 31 00:43:25.328478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.328452 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.349762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.349730 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww"] Oct 31 00:43:25.373448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.373413 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf88x\" (UniqueName: \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.373603 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.373563 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.373719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.373700 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.475114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.475024 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.475114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.475108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.475370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.475279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf88x\" (UniqueName: \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.476459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.476419 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.478267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.478233 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.496613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.496582 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf88x\" (UniqueName: \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\") pod \"maas-api-revoke-sub-0bc0ea72-dq5ww\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.656467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.656429 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:25.933460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.933369 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" event={"ID":"630e2837-390c-462b-9d0a-466237913999","Type":"ContainerDied","Data":"6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78"} Oct 31 00:43:25.933460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.933432 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a36548748dd41896678aaf17e61e40be9a842f756ab8284bf1b82018d990e78" Oct 31 00:43:25.933705 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.933692 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d" Oct 31 00:43:25.983044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:25.983006 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww"] Oct 31 00:43:25.987765 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:25.987699 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e1b1fa_4d94_4332_9f3b_9b588365d119.slice/crio-f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f WatchSource:0}: Error finding container f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f: Status 404 returned error can't find the container with id f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f Oct 31 00:43:26.268988 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.268917 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:26.269449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.269434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:26.270122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.270101 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:43:26.945523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.945489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" event={"ID":"13e1b1fa-4d94-4332-9f3b-9b588365d119","Type":"ContainerStarted","Data":"c6ae5e7f69b0055ba0db3d85c1916d4a530246fe81fcda01e8726718dc6505dc"} Oct 31 00:43:26.945711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.945550 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" event={"ID":"13e1b1fa-4d94-4332-9f3b-9b588365d119","Type":"ContainerStarted","Data":"f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f"} Oct 31 00:43:26.985631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:26.985585 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" podStartSLOduration=1.985569565 podStartE2EDuration="1.985569565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:26.983099433 +0000 UTC m=+1475.432187071" watchObservedRunningTime="2026-10-31 00:43:26.985569565 +0000 UTC m=+1475.434657221" Oct 31 00:43:27.957812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:27.957774 5011 generic.go:356] "Generic (PLEG): container finished" podID="13e1b1fa-4d94-4332-9f3b-9b588365d119" containerID="c6ae5e7f69b0055ba0db3d85c1916d4a530246fe81fcda01e8726718dc6505dc" exitCode=0 Oct 31 00:43:27.958353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:27.957857 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" event={"ID":"13e1b1fa-4d94-4332-9f3b-9b588365d119","Type":"ContainerDied","Data":"c6ae5e7f69b0055ba0db3d85c1916d4a530246fe81fcda01e8726718dc6505dc"} Oct 31 00:43:29.132036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.132006 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:29.245791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.245713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\" (UniqueName: \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\") pod \"13e1b1fa-4d94-4332-9f3b-9b588365d119\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " Oct 31 00:43:29.245791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.245763 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\") pod \"13e1b1fa-4d94-4332-9f3b-9b588365d119\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " Oct 31 00:43:29.246049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.245871 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\") pod \"13e1b1fa-4d94-4332-9f3b-9b588365d119\" (UID: \"13e1b1fa-4d94-4332-9f3b-9b588365d119\") " Oct 31 00:43:29.246480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.246283 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca" pod "13e1b1fa-4d94-4332-9f3b-9b588365d119" (UID: "13e1b1fa-4d94-4332-9f3b-9b588365d119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:29.246480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.246452 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:29.247889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.247864 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca" pod "13e1b1fa-4d94-4332-9f3b-9b588365d119" (UID: "13e1b1fa-4d94-4332-9f3b-9b588365d119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:29.248011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.247891 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x" pod "13e1b1fa-4d94-4332-9f3b-9b588365d119" (UID: "13e1b1fa-4d94-4332-9f3b-9b588365d119"). InnerVolumeSpecName "kube-api-access-kf88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:29.348046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.348011 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf88x\" (UniqueName: \"kubernetes.io/projected/13e1b1fa-4d94-4332-9f3b-9b588365d119-kube-api-access-kf88x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:29.348046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.348047 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e1b1fa-4d94-4332-9f3b-9b588365d119-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:29.975790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.975707 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" Oct 31 00:43:29.975790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.975743 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww" event={"ID":"13e1b1fa-4d94-4332-9f3b-9b588365d119","Type":"ContainerDied","Data":"f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f"} Oct 31 00:43:29.975790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:29.975790 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86f6f3a76343fc8aabe64b8a57c4f8bf04cc2818cda61b0991c92081069fe1f" Oct 31 00:43:31.965785 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:31.965748 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:43:32.406882 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.406838 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc"] Oct 31 00:43:32.407485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.407471 5011 cpu_manager.go:414] "containerMap: removing container" podUID="13e1b1fa-4d94-4332-9f3b-9b588365d119" containerName="revoke-keys" Oct 31 00:43:32.407544 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.407487 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e1b1fa-4d94-4332-9f3b-9b588365d119" containerName="revoke-keys" Oct 31 00:43:32.407593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.407584 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e1b1fa-4d94-4332-9f3b-9b588365d119" containerName="revoke-keys" Oct 31 00:43:32.411572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.411554 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.433248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.431139 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc"] Oct 31 00:43:32.434768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.434744 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:32.434899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.434869 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:32.497603 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.497574 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.497769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.497750 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.497896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.497883 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.599020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.598987 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.599166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.599155 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.599302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.599289 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.599821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.599798 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.601699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.601665 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.627339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.627314 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") pod \"maas-api-revoke-sub-b315f313-dwqxc\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:32.732388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:32.732291 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:33.099291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:33.098123 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc"] Oct 31 00:43:33.104792 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:33.104760 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd2117d_23db_4a81_962b_27e138695eaa.slice/crio-ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62 WatchSource:0}: Error finding container ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62: Status 404 returned error can't find the container with id ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62 Oct 31 00:43:34.120229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:34.115513 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" event={"ID":"9fd2117d-23db-4a81-962b-27e138695eaa","Type":"ContainerStarted","Data":"006e1a8ff4a33759d627e495f391266205b6bfdd230b571f2839364faa81a6e1"} Oct 31 00:43:34.120229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:34.115586 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" event={"ID":"9fd2117d-23db-4a81-962b-27e138695eaa","Type":"ContainerStarted","Data":"ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62"} Oct 31 00:43:34.221227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:34.205706 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" podStartSLOduration=2.205685671 podStartE2EDuration="2.205685671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:34.177892662 +0000 UTC m=+1482.626980302" watchObservedRunningTime="2026-10-31 00:43:34.205685671 +0000 UTC m=+1482.654773310" Oct 31 00:43:34.829338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:34.829291 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:34.895406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:34.884276 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.158934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.158842 5011 generic.go:356] "Generic (PLEG): container finished" podID="9fd2117d-23db-4a81-962b-27e138695eaa" containerID="006e1a8ff4a33759d627e495f391266205b6bfdd230b571f2839364faa81a6e1" exitCode=0 Oct 31 00:43:35.165661 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.165627 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" event={"ID":"9fd2117d-23db-4a81-962b-27e138695eaa","Type":"ContainerDied","Data":"006e1a8ff4a33759d627e495f391266205b6bfdd230b571f2839364faa81a6e1"} Oct 31 00:43:35.285543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.283306 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.285543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.283711 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.337232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.329485 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.360229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.343950 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.381414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.381322 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.404340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.404288 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.479473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.479367 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.503272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.502508 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.526437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.526394 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.629350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.629290 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.655379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.655338 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.872252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:35.869445 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.014698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.014651 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:36.015595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.015572 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:43:36.294524 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.294408 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.349236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.320062 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:43:36.349236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.323814 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.349236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.338475 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:43:36.357830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.357779 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.361353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.361330 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:43:36.479368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.479316 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:43:36.494238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.479940 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.512889 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.513251 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.513911 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.514232 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.558559 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.622299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.559054 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:43:36.881403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.815527 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:43:36.933992 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:36.816199 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:43:37.071232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.020184 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.261344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.074640 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.359850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.359787 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.380035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.379977 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.455344 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.429758 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:43:37.486472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.486423 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:43:37.604167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.604117 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.647054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.624054 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:43:37.647054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.634939 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.647054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.635292 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.647054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.636612 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 31 00:43:37.647054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.636669 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:37.674570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.674528 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.847969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.838475 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:43:37.927288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:37.926252 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.467231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.406314 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.467231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.407354 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:38.467231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.440307 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.606624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.586907 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.606624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.604671 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.632360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.607119 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:43:38.657287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.608987 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.678231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.631740 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.678231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.646603 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:43:38.678231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.646642 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.678231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.646668 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.687248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.679022 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:43:38.687248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.679099 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:43:38.718428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.718309 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.891037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.866329 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.936607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.927681 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.941682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:38.941654 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:43:39.234281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.232614 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 31 00:43:39.245229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.235165 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 31 00:43:39.301624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.301580 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 00:43:39.548449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.548348 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:43:39.595250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.548515 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:39.678364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.678312 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.712179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.712130 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:43:39.839905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.772877 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.109712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.817929 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:43:40.110194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.817957 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.110564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.819020 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.110890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.819044 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.111043 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.873511 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:43:40.115226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.881332 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.131236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.881970 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:43:40.187295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.963609 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.187295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:39.978185 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.187295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.174562 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.215374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.190920 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:40.225480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.225447 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk"] Oct 31 00:43:40.250226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.246136 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9fd2117d-23db-4a81-962b-27e138695eaa" containerName="revoke-keys" Oct 31 00:43:40.279300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.279254 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd2117d-23db-4a81-962b-27e138695eaa" containerName="revoke-keys" Oct 31 00:43:40.279852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.279829 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd2117d-23db-4a81-962b-27e138695eaa" containerName="revoke-keys" Oct 31 00:43:40.305445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.305415 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:43:40.344724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.338919 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk"] Oct 31 00:43:40.344724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.339094 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.344724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.343196 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") pod \"9fd2117d-23db-4a81-962b-27e138695eaa\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " Oct 31 00:43:40.344724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.343313 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\") pod \"9fd2117d-23db-4a81-962b-27e138695eaa\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " Oct 31 00:43:40.344724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.343509 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\") pod \"9fd2117d-23db-4a81-962b-27e138695eaa\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " Oct 31 00:43:40.386658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.378056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca" pod "9fd2117d-23db-4a81-962b-27e138695eaa" (UID: "9fd2117d-23db-4a81-962b-27e138695eaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:40.386658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.378158 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 31 00:43:40.386658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.378511 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:40.531231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.499364 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.568229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.544266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.590243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.569093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pfw\" (UniqueName: \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.740351 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.758057 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.758411 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.778532 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.778796 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pfw\" (UniqueName: \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.779875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.778874 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.807837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.805445 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" event={"ID":"9fd2117d-23db-4a81-962b-27e138695eaa","Type":"ContainerDied","Data":"ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62"} Oct 31 00:43:40.807837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.805509 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae44cfd533302102f0377445968859c060ddec060d23274650ec8ccf7c2c6d62" Oct 31 00:43:40.807837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.805591 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc" Oct 31 00:43:40.827713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.827681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.872228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.858682 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.927245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.927191 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pfw\" (UniqueName: \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\") pod \"maas-api-revoke-sub-a8773200-bj8zk\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:40.955861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:40.945232 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:43:41.016266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.014475 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca" pod "9fd2117d-23db-4a81-962b-27e138695eaa" (UID: "9fd2117d-23db-4a81-962b-27e138695eaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:41.030572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.030470 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fd2117d-23db-4a81-962b-27e138695eaa-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:41.115180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.114012 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.115180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.114397 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:43:41.115180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.114683 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:43:41.115180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.114932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.138283 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.140233 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.158582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5" pod "9fd2117d-23db-4a81-962b-27e138695eaa" (UID: "9fd2117d-23db-4a81-962b-27e138695eaa"). InnerVolumeSpecName "kube-api-access-rgbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.174086 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") pod \"9fd2117d-23db-4a81-962b-27e138695eaa\" (UID: \"9fd2117d-23db-4a81-962b-27e138695eaa\") " Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:41.184378 5011 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/9fd2117d-23db-4a81-962b-27e138695eaa/volumes/kubernetes.io~projected/kube-api-access-rgbs5 Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.184449 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5" pod "9fd2117d-23db-4a81-962b-27e138695eaa" (UID: "9fd2117d-23db-4a81-962b-27e138695eaa"). InnerVolumeSpecName "kube-api-access-rgbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.192926 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/9fd2117d-23db-4a81-962b-27e138695eaa-kube-api-access-rgbs5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.201022 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.236700 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.239163 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.256293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.240463 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.269233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.260723 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:43:41.284277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.270724 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:43:41.285219 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.285174 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:43:41.512304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.502460 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:43:41.512743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.512682 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.610808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.610676 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:43:41.705708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.705680 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:43:41.776680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.776642 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.809219 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.807624 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:43:41.809219 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.807952 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951481 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951544 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951577 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951609 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951700 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951734 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:43:41.955180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:41.951805 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:43:42.084939 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:42.083777 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk"] Oct 31 00:43:42.948302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:42.948259 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" event={"ID":"2a030669-82b1-4d51-bb50-dc869dfe19d0","Type":"ContainerStarted","Data":"93d6d7244ad3f3b6b112b6113a4f0bb4ecea2f1e67cfb7003c1de4dbdec84e5a"} Oct 31 00:43:42.948850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:42.948332 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" event={"ID":"2a030669-82b1-4d51-bb50-dc869dfe19d0","Type":"ContainerStarted","Data":"ebea834c80e86c1a72c578cedaea746b2a7048d7222e7c3476f5b53b10352c16"} Oct 31 00:43:42.996403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:42.996352 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" podStartSLOduration=3.996337991 podStartE2EDuration="3.996337991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:42.992510499 +0000 UTC m=+1491.441598141" watchObservedRunningTime="2026-10-31 00:43:42.996337991 +0000 UTC m=+1491.445425630" Oct 31 00:43:43.177463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.177433 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47"] Oct 31 00:43:43.182021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.181998 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.199849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.199793 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47"] Oct 31 00:43:43.297905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.297871 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.298066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.297958 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.298119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.298100 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhzh\" (UniqueName: \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.399526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.399491 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.399710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.399613 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhzh\" (UniqueName: \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.399796 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.399783 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.400294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.400242 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.402270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.402248 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.422014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.421973 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhzh\" (UniqueName: \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\") pod \"maas-api-revoke-sub-6b2e107f-96x47\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.497700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.497626 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:43.821894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.821863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47"] Oct 31 00:43:43.826651 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:43.826622 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c808da_0d3e_4797_ab28_a19788b42fa9.slice/crio-1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2 WatchSource:0}: Error finding container 1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2: Status 404 returned error can't find the container with id 1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2 Oct 31 00:43:43.961752 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.961717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" event={"ID":"69c808da-0d3e-4797-ab28-a19788b42fa9","Type":"ContainerStarted","Data":"1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2"} Oct 31 00:43:43.963856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.963825 5011 generic.go:356] "Generic (PLEG): container finished" podID="2a030669-82b1-4d51-bb50-dc869dfe19d0" containerID="93d6d7244ad3f3b6b112b6113a4f0bb4ecea2f1e67cfb7003c1de4dbdec84e5a" exitCode=0 Oct 31 00:43:43.963979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:43.963913 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" event={"ID":"2a030669-82b1-4d51-bb50-dc869dfe19d0","Type":"ContainerDied","Data":"93d6d7244ad3f3b6b112b6113a4f0bb4ecea2f1e67cfb7003c1de4dbdec84e5a"} Oct 31 00:43:44.977949 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:44.977864 5011 generic.go:356] "Generic (PLEG): container finished" podID="69c808da-0d3e-4797-ab28-a19788b42fa9" containerID="f01c2dfd9a5f36327f6511cbaf0f2574c81b0ef4b11c8dfa17bf339bd53dd6c8" exitCode=0 Oct 31 00:43:44.978340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:44.977959 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" event={"ID":"69c808da-0d3e-4797-ab28-a19788b42fa9","Type":"ContainerDied","Data":"f01c2dfd9a5f36327f6511cbaf0f2574c81b0ef4b11c8dfa17bf339bd53dd6c8"} Oct 31 00:43:45.162111 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.162075 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:45.233738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.233659 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\" (UniqueName: \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\") pod \"2a030669-82b1-4d51-bb50-dc869dfe19d0\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " Oct 31 00:43:45.233970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.233949 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\") pod \"2a030669-82b1-4d51-bb50-dc869dfe19d0\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " Oct 31 00:43:45.234057 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.234006 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\") pod \"2a030669-82b1-4d51-bb50-dc869dfe19d0\" (UID: \"2a030669-82b1-4d51-bb50-dc869dfe19d0\") " Oct 31 00:43:45.235334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.235305 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca" pod "2a030669-82b1-4d51-bb50-dc869dfe19d0" (UID: "2a030669-82b1-4d51-bb50-dc869dfe19d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:45.236953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.236922 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca" pod "2a030669-82b1-4d51-bb50-dc869dfe19d0" (UID: "2a030669-82b1-4d51-bb50-dc869dfe19d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:45.237181 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.237159 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw" pod "2a030669-82b1-4d51-bb50-dc869dfe19d0" (UID: "2a030669-82b1-4d51-bb50-dc869dfe19d0"). InnerVolumeSpecName "kube-api-access-j7pfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:45.335952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.335915 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pfw\" (UniqueName: \"kubernetes.io/projected/2a030669-82b1-4d51-bb50-dc869dfe19d0-kube-api-access-j7pfw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:45.335952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.335951 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:45.336166 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.335966 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a030669-82b1-4d51-bb50-dc869dfe19d0-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:45.988351 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.988267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" Oct 31 00:43:45.988727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.988271 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk" event={"ID":"2a030669-82b1-4d51-bb50-dc869dfe19d0","Type":"ContainerDied","Data":"ebea834c80e86c1a72c578cedaea746b2a7048d7222e7c3476f5b53b10352c16"} Oct 31 00:43:45.988727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:45.988433 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebea834c80e86c1a72c578cedaea746b2a7048d7222e7c3476f5b53b10352c16" Oct 31 00:43:46.164355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.164329 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:46.249092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.249016 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\") pod \"69c808da-0d3e-4797-ab28-a19788b42fa9\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " Oct 31 00:43:46.249289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.249153 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\" (UniqueName: \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\") pod \"69c808da-0d3e-4797-ab28-a19788b42fa9\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " Oct 31 00:43:46.249289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.249255 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\") pod \"69c808da-0d3e-4797-ab28-a19788b42fa9\" (UID: \"69c808da-0d3e-4797-ab28-a19788b42fa9\") " Oct 31 00:43:46.249693 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.249660 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca" pod "69c808da-0d3e-4797-ab28-a19788b42fa9" (UID: "69c808da-0d3e-4797-ab28-a19788b42fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:46.250062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.250032 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.251462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.251432 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca" pod "69c808da-0d3e-4797-ab28-a19788b42fa9" (UID: "69c808da-0d3e-4797-ab28-a19788b42fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:46.251556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.251523 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh" pod "69c808da-0d3e-4797-ab28-a19788b42fa9" (UID: "69c808da-0d3e-4797-ab28-a19788b42fa9"). InnerVolumeSpecName "kube-api-access-6hhzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:46.351053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.351025 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hhzh\" (UniqueName: \"kubernetes.io/projected/69c808da-0d3e-4797-ab28-a19788b42fa9-kube-api-access-6hhzh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.351053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.351052 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c808da-0d3e-4797-ab28-a19788b42fa9-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.926741 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.926703 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj"] Oct 31 00:43:46.927626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927603 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2a030669-82b1-4d51-bb50-dc869dfe19d0" containerName="revoke-keys" Oct 31 00:43:46.927696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927630 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a030669-82b1-4d51-bb50-dc869dfe19d0" containerName="revoke-keys" Oct 31 00:43:46.927696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927689 5011 cpu_manager.go:414] "containerMap: removing container" podUID="69c808da-0d3e-4797-ab28-a19788b42fa9" containerName="revoke-keys" Oct 31 00:43:46.927762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927697 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c808da-0d3e-4797-ab28-a19788b42fa9" containerName="revoke-keys" Oct 31 00:43:46.927872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927861 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c808da-0d3e-4797-ab28-a19788b42fa9" containerName="revoke-keys" Oct 31 00:43:46.927904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.927884 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a030669-82b1-4d51-bb50-dc869dfe19d0" containerName="revoke-keys" Oct 31 00:43:46.932152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.932136 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:46.949399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.949369 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj"] Oct 31 00:43:46.961469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.961443 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:46.961720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.961693 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw2d2\" (UniqueName: \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:46.961839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.961742 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:46.999401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.999373 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" Oct 31 00:43:46.999786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.999410 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47" event={"ID":"69c808da-0d3e-4797-ab28-a19788b42fa9","Type":"ContainerDied","Data":"1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2"} Oct 31 00:43:46.999786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:46.999463 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1982f388df03f557d7401c63e49bf51b8f8f13c0d7ad9d9049ec328fb57757d2" Oct 31 00:43:47.063676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.063640 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.063825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.063739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw2d2\" (UniqueName: \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.063825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.063765 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.064340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.064313 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.066222 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.066190 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.085246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.085226 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw2d2\" (UniqueName: \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\") pod \"maas-api-revoke-sub-0be39919-2znqj\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.247073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.246988 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:47.553844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.553817 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj"] Oct 31 00:43:47.553844 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:47.553829 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62198c3a_093b_45e4_8b86_324b35e6c5ff.slice/crio-5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5 WatchSource:0}: Error finding container 5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5: Status 404 returned error can't find the container with id 5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5 Oct 31 00:43:47.671072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.671043 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:43:47.676716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.676692 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.699218 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.699181 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:43:47.773353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.773313 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.773532 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.773465 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774070 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774125 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774262 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774305 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774395 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774425 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.774523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.774472 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbm5\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.876485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbm5\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.876650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.876650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876615 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.876773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876666 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.876773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.877522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.877432 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.877522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.876787 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.877842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.877771 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.877990 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.878065 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.878090 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.878127 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.878382 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.878859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.878737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.879824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.879791 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.888953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.888930 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.898462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.898436 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbm5\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.900141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.900120 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:47.996006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:47.995977 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:48.009244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.009190 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" event={"ID":"62198c3a-093b-45e4-8b86-324b35e6c5ff","Type":"ContainerStarted","Data":"d2e584d4ebe4f2e9196bb8c11d6148bb406bead176a846b91e48d9133ecea169"} Oct 31 00:43:48.009535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.009278 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" event={"ID":"62198c3a-093b-45e4-8b86-324b35e6c5ff","Type":"ContainerStarted","Data":"5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5"} Oct 31 00:43:48.057908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.054313 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" podStartSLOduration=2.054293392 podStartE2EDuration="2.054293392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:48.050012547 +0000 UTC m=+1496.499100178" watchObservedRunningTime="2026-10-31 00:43:48.054293392 +0000 UTC m=+1496.503381034" Oct 31 00:43:48.337046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.337011 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:43:48.338023 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:48.337992 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408e0ec0_7e05_4a55_84d9_4ea5d2c7b557.slice/crio-aae52f3185d502f0c20560bd09afb981bd674f1bf7c5559b1f61bcbae25679d9 WatchSource:0}: Error finding container aae52f3185d502f0c20560bd09afb981bd674f1bf7c5559b1f61bcbae25679d9: Status 404 returned error can't find the container with id aae52f3185d502f0c20560bd09afb981bd674f1bf7c5559b1f61bcbae25679d9 Oct 31 00:43:48.354413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.354373 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:48.354623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.354471 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:48.354696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:48.354639 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:43:49.044568 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.044526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" event={"ID":"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557","Type":"ContainerStarted","Data":"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856"} Oct 31 00:43:49.045083 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.044598 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" event={"ID":"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557","Type":"ContainerStarted","Data":"aae52f3185d502f0c20560bd09afb981bd674f1bf7c5559b1f61bcbae25679d9"} Oct 31 00:43:49.048242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.048171 5011 generic.go:356] "Generic (PLEG): container finished" podID="62198c3a-093b-45e4-8b86-324b35e6c5ff" containerID="d2e584d4ebe4f2e9196bb8c11d6148bb406bead176a846b91e48d9133ecea169" exitCode=0 Oct 31 00:43:49.048386 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.048257 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" event={"ID":"62198c3a-093b-45e4-8b86-324b35e6c5ff","Type":"ContainerDied","Data":"d2e584d4ebe4f2e9196bb8c11d6148bb406bead176a846b91e48d9133ecea169"} Oct 31 00:43:49.112167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.112116 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podStartSLOduration=2.112102971 podStartE2EDuration="2.112102971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:49.09797778 +0000 UTC m=+1497.547065420" watchObservedRunningTime="2026-10-31 00:43:49.112102971 +0000 UTC m=+1497.561190672" Oct 31 00:43:49.997134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.997094 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:49.999142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.999114 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:49.999301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:49.999164 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:50.315992 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.315544 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:50.464608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.464574 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\" (UniqueName: \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\") pod \"62198c3a-093b-45e4-8b86-324b35e6c5ff\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " Oct 31 00:43:50.464776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.464627 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\") pod \"62198c3a-093b-45e4-8b86-324b35e6c5ff\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " Oct 31 00:43:50.464776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.464693 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\") pod \"62198c3a-093b-45e4-8b86-324b35e6c5ff\" (UID: \"62198c3a-093b-45e4-8b86-324b35e6c5ff\") " Oct 31 00:43:50.465630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.465587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca" pod "62198c3a-093b-45e4-8b86-324b35e6c5ff" (UID: "62198c3a-093b-45e4-8b86-324b35e6c5ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:50.468965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.468865 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca" pod "62198c3a-093b-45e4-8b86-324b35e6c5ff" (UID: "62198c3a-093b-45e4-8b86-324b35e6c5ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:50.468965 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.468923 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2" pod "62198c3a-093b-45e4-8b86-324b35e6c5ff" (UID: "62198c3a-093b-45e4-8b86-324b35e6c5ff"). InnerVolumeSpecName "kube-api-access-xw2d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:50.551614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.551541 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10899-a21868"] Oct 31 00:43:50.552201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.552185 5011 cpu_manager.go:414] "containerMap: removing container" podUID="62198c3a-093b-45e4-8b86-324b35e6c5ff" containerName="revoke-keys" Oct 31 00:43:50.552272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.552217 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62198c3a-093b-45e4-8b86-324b35e6c5ff" containerName="revoke-keys" Oct 31 00:43:50.552400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.552389 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="62198c3a-093b-45e4-8b86-324b35e6c5ff" containerName="revoke-keys" Oct 31 00:43:50.556119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.556099 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:50.566901 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.566871 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw2d2\" (UniqueName: \"kubernetes.io/projected/62198c3a-093b-45e4-8b86-324b35e6c5ff-kube-api-access-xw2d2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:50.567010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.566907 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:50.567010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.566922 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62198c3a-093b-45e4-8b86-324b35e6c5ff-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:50.567615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.567594 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10899-a21868"] Oct 31 00:43:50.569218 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.569183 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\"" Oct 31 00:43:50.668312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.668279 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl67w\" (UniqueName: \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\") pod \"test-curl-10899-a21868\" (UID: \"2acad693-30c6-403a-9c24-b98e54bb70c1\") " pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:50.770063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.770025 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl67w\" (UniqueName: \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\") pod \"test-curl-10899-a21868\" (UID: \"2acad693-30c6-403a-9c24-b98e54bb70c1\") " pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:50.792331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.792301 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl67w\" (UniqueName: \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\") pod \"test-curl-10899-a21868\" (UID: \"2acad693-30c6-403a-9c24-b98e54bb70c1\") " pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:50.873691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.873651 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:50.997423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.997386 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:51.000331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:50.997447 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:51.067991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.067950 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" event={"ID":"62198c3a-093b-45e4-8b86-324b35e6c5ff","Type":"ContainerDied","Data":"5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5"} Oct 31 00:43:51.068152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.068038 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b28a5a526010aedd4f11fc329120eb0fb71bd8d7ecbcf87cac0e59ae4dd86c5" Oct 31 00:43:51.068152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.067965 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj" Oct 31 00:43:51.211630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.211603 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10899-a21868"] Oct 31 00:43:51.212496 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:51.212471 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2acad693_30c6_403a_9c24_b98e54bb70c1.slice/crio-ff95cd28d8a96428505276f43dcd22a1501caaebac52e680fdb95e8a17f1ac76 WatchSource:0}: Error finding container ff95cd28d8a96428505276f43dcd22a1501caaebac52e680fdb95e8a17f1ac76: Status 404 returned error can't find the container with id ff95cd28d8a96428505276f43dcd22a1501caaebac52e680fdb95e8a17f1ac76 Oct 31 00:43:51.996754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.996559 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:51.996754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:51.996617 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:52.007241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.007180 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:43:52.052201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.052114 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:43:52.078517 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.078478 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10899-a21868" event={"ID":"2acad693-30c6-403a-9c24-b98e54bb70c1","Type":"ContainerStarted","Data":"c49ba0be99d0ba0d5d228f2d283d5cfe08003b00906c944ae3549f75ffb336f7"} Oct 31 00:43:52.078667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.078533 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10899-a21868" event={"ID":"2acad693-30c6-403a-9c24-b98e54bb70c1","Type":"ContainerStarted","Data":"ff95cd28d8a96428505276f43dcd22a1501caaebac52e680fdb95e8a17f1ac76"} Oct 31 00:43:52.115830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.115776 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10899-a21868" podStartSLOduration=1.564861115 podStartE2EDuration="2.115761349s" totalImagesPullingTime="550.900234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:52.114560636 +0000 UTC m=+1500.563648276" watchObservedRunningTime="2026-10-31 00:43:52.115761349 +0000 UTC m=+1500.564848988" Oct 31 00:43:52.996902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.996866 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:52.997385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:52.996927 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:53.084884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.084845 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10899-a21868"] Oct 31 00:43:53.090185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.090151 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10899-a21868"] Oct 31 00:43:53.258324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.258239 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10899-613e47"] Oct 31 00:43:53.258896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.258878 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" containerName="test-curl-10899-a21868" Oct 31 00:43:53.258977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.258896 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" containerName="test-curl-10899-a21868" Oct 31 00:43:53.259049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.259035 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" containerName="test-curl-10899-a21868" Oct 31 00:43:53.263129 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.263104 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:53.286675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.286648 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10899-613e47"] Oct 31 00:43:53.414125 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.414085 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7sg6\" (UniqueName: \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\") pod \"test-curl-10899-613e47\" (UID: \"1b1b5fc5-f871-42c0-b1b9-0f50142229eb\") " pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:53.515224 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.515132 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7sg6\" (UniqueName: \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\") pod \"test-curl-10899-613e47\" (UID: \"1b1b5fc5-f871-42c0-b1b9-0f50142229eb\") " pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:53.545888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.545851 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7sg6\" (UniqueName: \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\") pod \"test-curl-10899-613e47\" (UID: \"1b1b5fc5-f871-42c0-b1b9-0f50142229eb\") " pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:53.577538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.577513 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:53.711980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.711821 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg"] Oct 31 00:43:53.751265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.749519 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.771750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.768619 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:53.781908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.781879 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg"] Oct 31 00:43:53.857320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.857271 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.857809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.857780 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.858050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.858032 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcrr\" (UniqueName: \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.872298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.865751 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:53.946436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.946040 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10899-613e47"] Oct 31 00:43:53.957649 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:53.954418 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b1b5fc5_f871_42c0_b1b9_0f50142229eb.slice/crio-79830bbaa9ac4d8c86971ccf13e96abd091b701eb236cbb5f03c1b7d024343a4 WatchSource:0}: Error finding container 79830bbaa9ac4d8c86971ccf13e96abd091b701eb236cbb5f03c1b7d024343a4: Status 404 returned error can't find the container with id 79830bbaa9ac4d8c86971ccf13e96abd091b701eb236cbb5f03c1b7d024343a4 Oct 31 00:43:53.963075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.959195 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcrr\" (UniqueName: \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.963075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.960856 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.963075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.961019 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.963075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.961910 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.967539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.967515 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.984856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.984825 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcrr\" (UniqueName: \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\") pod \"maas-api-revoke-sub-9f4b9d96-k2wwg\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:53.997448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.997415 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:53.997871 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:53.997511 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:54.087609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.087578 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:54.102272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.102238 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10899-613e47" event={"ID":"1b1b5fc5-f871-42c0-b1b9-0f50142229eb","Type":"ContainerStarted","Data":"983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426"} Oct 31 00:43:54.102377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.102289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10899-613e47" event={"ID":"1b1b5fc5-f871-42c0-b1b9-0f50142229eb","Type":"ContainerStarted","Data":"79830bbaa9ac4d8c86971ccf13e96abd091b701eb236cbb5f03c1b7d024343a4"} Oct 31 00:43:54.102551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.102519 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10899-a21868" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" containerName="test-curl-10899-a21868" containerID="cri-o://c49ba0be99d0ba0d5d228f2d283d5cfe08003b00906c944ae3549f75ffb336f7" gracePeriod=2 Oct 31 00:43:54.156111 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.156055 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10899-613e47" podStartSLOduration=1.1560397550000001 podStartE2EDuration="1.156039755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:54.144512926 +0000 UTC m=+1502.593600567" watchObservedRunningTime="2026-10-31 00:43:54.156039755 +0000 UTC m=+1502.605127393" Oct 31 00:43:54.444077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.444045 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg"] Oct 31 00:43:54.445149 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:54.445119 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeff3e32_1c27_4d0f_a9a1_7ec5fba1ca6d.slice/crio-bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105 WatchSource:0}: Error finding container bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105: Status 404 returned error can't find the container with id bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105 Oct 31 00:43:54.996270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.996186 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:43:54.996270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:54.996250 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 00:43:55.117171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:55.117115 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" event={"ID":"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d","Type":"ContainerStarted","Data":"5b53bfcf3b160f3546bab04a37299242950f04bed0c3cc24a81fa34963998dfa"} Oct 31 00:43:55.117663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:55.117195 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" event={"ID":"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d","Type":"ContainerStarted","Data":"bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105"} Oct 31 00:43:55.160709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:55.160677 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10899-613e47"] Oct 31 00:43:55.205479 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:55.205442 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10899-613e47"] Oct 31 00:43:56.000423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.000392 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:56.000908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.000886 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:56.001760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.001740 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:43:56.067005 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.066954 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" podStartSLOduration=3.066935212 podStartE2EDuration="3.066935212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:55.201535307 +0000 UTC m=+1503.650622946" watchObservedRunningTime="2026-10-31 00:43:56.066935212 +0000 UTC m=+1504.516022851" Oct 31 00:43:56.149299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.149261 5011 generic.go:356] "Generic (PLEG): container finished" podID="2acad693-30c6-403a-9c24-b98e54bb70c1" containerID="c49ba0be99d0ba0d5d228f2d283d5cfe08003b00906c944ae3549f75ffb336f7" exitCode=137 Oct 31 00:43:56.152022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.151552 5011 generic.go:356] "Generic (PLEG): container finished" podID="feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" containerID="5b53bfcf3b160f3546bab04a37299242950f04bed0c3cc24a81fa34963998dfa" exitCode=0 Oct 31 00:43:56.152900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.152863 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" event={"ID":"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d","Type":"ContainerDied","Data":"5b53bfcf3b160f3546bab04a37299242950f04bed0c3cc24a81fa34963998dfa"} Oct 31 00:43:56.153052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.153001 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10899-613e47" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" containerName="test-curl-10899-613e47" containerID="cri-o://983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426" gracePeriod=2 Oct 31 00:43:56.380020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.379992 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:56.405381 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.405356 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\" (UniqueName: \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\") pod \"2acad693-30c6-403a-9c24-b98e54bb70c1\" (UID: \"2acad693-30c6-403a-9c24-b98e54bb70c1\") " Oct 31 00:43:56.407461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.407433 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w" pod "2acad693-30c6-403a-9c24-b98e54bb70c1" (UID: "2acad693-30c6-403a-9c24-b98e54bb70c1"). InnerVolumeSpecName "kube-api-access-nl67w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:56.507053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:56.507022 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl67w\" (UniqueName: \"kubernetes.io/projected/2acad693-30c6-403a-9c24-b98e54bb70c1-kube-api-access-nl67w\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:57.161506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.161477 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:57.161981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.161521 5011 scope.go:122] "RemoveContainer" containerID="c49ba0be99d0ba0d5d228f2d283d5cfe08003b00906c944ae3549f75ffb336f7" Oct 31 00:43:57.184468 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:57.184429 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10899-a21868\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:57.338297 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.338269 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:57.388733 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:57.388599 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10899-a21868\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" pod="opendatahub/test-curl-10899-a21868" Oct 31 00:43:57.426042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.420351 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\") pod \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " Oct 31 00:43:57.426042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.420564 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\" (UniqueName: \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\") pod \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " Oct 31 00:43:57.426042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.420647 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\") pod \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\" (UID: \"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d\") " Oct 31 00:43:57.426714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.426683 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca" pod "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" (UID: "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.427177 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.427144 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:57.429036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.429007 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr" pod "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" (UID: "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d"). InnerVolumeSpecName "kube-api-access-wbcrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.429585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.429563 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca" pod "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" (UID: "feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.529333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.529283 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:57.529333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.529327 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbcrr\" (UniqueName: \"kubernetes.io/projected/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d-kube-api-access-wbcrr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:57.684851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:57.684773 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acad693-30c6-403a-9c24-b98e54bb70c1" path="/var/lib/kubelet/pods/2acad693-30c6-403a-9c24-b98e54bb70c1/volumes" Oct 31 00:43:58.173226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.173181 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" Oct 31 00:43:58.173673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.173424 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg" event={"ID":"feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d","Type":"ContainerDied","Data":"bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105"} Oct 31 00:43:58.173673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.173471 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfaa7b5ac2552fa445f3a22b98dc2788b8dc544804d58e4e04648ed63b0fe105" Oct 31 00:43:58.308419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.308387 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8"] Oct 31 00:43:58.309356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309334 5011 cpu_manager.go:414] "containerMap: removing container" podUID="feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" containerName="revoke-keys" Oct 31 00:43:58.309356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309354 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" containerName="revoke-keys" Oct 31 00:43:58.309514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309365 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" containerName="test-curl-10899-613e47" Oct 31 00:43:58.309514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309371 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" containerName="test-curl-10899-613e47" Oct 31 00:43:58.309514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309504 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" containerName="test-curl-10899-613e47" Oct 31 00:43:58.309676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.309522 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" containerName="revoke-keys" Oct 31 00:43:58.313179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.313159 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.335324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.333566 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:58.335324 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.334876 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8"] Oct 31 00:43:58.337738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.337717 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:58.359370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.359343 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9p2\" (UniqueName: \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.359558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.359533 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.359678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.359658 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.413153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.413090 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:58.461421 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.461387 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\" (UniqueName: \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\") pod \"1b1b5fc5-f871-42c0-b1b9-0f50142229eb\" (UID: \"1b1b5fc5-f871-42c0-b1b9-0f50142229eb\") " Oct 31 00:43:58.461795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.461774 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9p2\" (UniqueName: \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.461901 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.461886 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.461959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.461940 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.462807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.462781 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.463712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.463687 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6" pod "1b1b5fc5-f871-42c0-b1b9-0f50142229eb" (UID: "1b1b5fc5-f871-42c0-b1b9-0f50142229eb"). InnerVolumeSpecName "kube-api-access-p7sg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.464407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.464380 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.490432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.490406 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9p2\" (UniqueName: \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\") pod \"maas-api-revoke-sub-c5813c6c-mw6r8\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.563676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.563653 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7sg6\" (UniqueName: \"kubernetes.io/projected/1b1b5fc5-f871-42c0-b1b9-0f50142229eb-kube-api-access-p7sg6\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:43:58.639323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.639292 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:43:58.956685 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:43:58.956641 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a556e5d_f3a8_43a4_b325_96a116ef5555.slice/crio-f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1 WatchSource:0}: Error finding container f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1: Status 404 returned error can't find the container with id f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1 Oct 31 00:43:58.957265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:58.957240 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8"] Oct 31 00:43:59.183993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.183957 5011 generic.go:356] "Generic (PLEG): container finished" podID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" containerID="983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426" exitCode=137 Oct 31 00:43:59.184459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.183997 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:59.184459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.184014 5011 scope.go:122] "RemoveContainer" containerID="983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426" Oct 31 00:43:59.185904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.185882 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" event={"ID":"2a556e5d-f3a8-43a4-b325-96a116ef5555","Type":"ContainerStarted","Data":"f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1"} Oct 31 00:43:59.196059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.195889 5011 scope.go:122] "RemoveContainer" containerID="983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426" Oct 31 00:43:59.196158 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:59.196141 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426\": container with ID starting with 983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426 not found: ID does not exist" containerID="983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426" Oct 31 00:43:59.196221 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.196168 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426"} err="failed to get container status \"983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426\": rpc error: code = NotFound desc = could not find container \"983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426\": container with ID starting with 983cc56d17db4ee0725d90d65baf86b49ccc8186dc828a598c14bedd6912c426 not found: ID does not exist" Oct 31 00:43:59.206004 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:43:59.205953 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10899-613e47\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" pod="opendatahub/test-curl-10899-613e47" Oct 31 00:43:59.683823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:43:59.683791 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1b5fc5-f871-42c0-b1b9-0f50142229eb" path="/var/lib/kubelet/pods/1b1b5fc5-f871-42c0-b1b9-0f50142229eb/volumes" Oct 31 00:44:00.198081 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:00.198046 5011 generic.go:356] "Generic (PLEG): container finished" podID="2a556e5d-f3a8-43a4-b325-96a116ef5555" containerID="fbd5bd482ab51110994af28af52191795f3f8b34c5e71336181d4a01d425b441" exitCode=0 Oct 31 00:44:00.198511 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:00.198123 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" event={"ID":"2a556e5d-f3a8-43a4-b325-96a116ef5555","Type":"ContainerDied","Data":"fbd5bd482ab51110994af28af52191795f3f8b34c5e71336181d4a01d425b441"} Oct 31 00:44:01.394054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.394026 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:44:01.504923 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.504849 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\") pod \"2a556e5d-f3a8-43a4-b325-96a116ef5555\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " Oct 31 00:44:01.505056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.504964 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\") pod \"2a556e5d-f3a8-43a4-b325-96a116ef5555\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " Oct 31 00:44:01.505056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.505050 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\" (UniqueName: \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\") pod \"2a556e5d-f3a8-43a4-b325-96a116ef5555\" (UID: \"2a556e5d-f3a8-43a4-b325-96a116ef5555\") " Oct 31 00:44:01.505355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.505327 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca" pod "2a556e5d-f3a8-43a4-b325-96a116ef5555" (UID: "2a556e5d-f3a8-43a4-b325-96a116ef5555"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:01.505469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.505449 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:01.507080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.507060 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca" pod "2a556e5d-f3a8-43a4-b325-96a116ef5555" (UID: "2a556e5d-f3a8-43a4-b325-96a116ef5555"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:01.507139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.507093 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2" pod "2a556e5d-f3a8-43a4-b325-96a116ef5555" (UID: "2a556e5d-f3a8-43a4-b325-96a116ef5555"). InnerVolumeSpecName "kube-api-access-rx9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:01.606628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.606593 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a556e5d-f3a8-43a4-b325-96a116ef5555-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:01.606628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:01.606623 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9p2\" (UniqueName: \"kubernetes.io/projected/2a556e5d-f3a8-43a4-b325-96a116ef5555-kube-api-access-rx9p2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:01.948348 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:01.948310 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:44:02.215457 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:02.215372 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" Oct 31 00:44:02.215625 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:02.215376 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8" event={"ID":"2a556e5d-f3a8-43a4-b325-96a116ef5555","Type":"ContainerDied","Data":"f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1"} Oct 31 00:44:02.215625 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:02.215507 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8091879252bb1ab4ad275c6c2e7230f6f5563134429d76ce2f8615a4dc4ead1" Oct 31 00:44:03.597500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.597465 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:44:03.598141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.598123 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2a556e5d-f3a8-43a4-b325-96a116ef5555" containerName="revoke-keys" Oct 31 00:44:03.598193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.598144 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a556e5d-f3a8-43a4-b325-96a116ef5555" containerName="revoke-keys" Oct 31 00:44:03.598278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.598268 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a556e5d-f3a8-43a4-b325-96a116ef5555" containerName="revoke-keys" Oct 31 00:44:03.601482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.601466 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.643634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.643600 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:44:03.741856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.741816 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.741856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.741857 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.741890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887h2\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742039 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742075 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742168 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742246 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742309 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.742503 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.742479 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844010 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844238 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844269 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887h2\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844309 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844337 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844412 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844465 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.844617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.844498 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.845573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.845276 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.845573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.845301 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.845573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.845495 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.845822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.845606 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.845822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.845759 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.847153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.847126 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.847404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.847384 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.867619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.867554 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.867784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.867767 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887h2\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:03.917409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:03.917378 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:04.266110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:04.266009 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:44:04.271135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:04.271109 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:44:04.271369 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:04.271341 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b9146d_33e4_4a0b_a3ec_8c69f051e30d.slice/crio-e398e60f37eab459cd90c59aab30950a4d61933ff2fb07d04e4fd28088e3a8d7 WatchSource:0}: Error finding container e398e60f37eab459cd90c59aab30950a4d61933ff2fb07d04e4fd28088e3a8d7: Status 404 returned error can't find the container with id e398e60f37eab459cd90c59aab30950a4d61933ff2fb07d04e4fd28088e3a8d7 Oct 31 00:44:04.273881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:04.273849 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:04.273961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:04.273910 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:04.273961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:04.273938 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:05.198301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.198266 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6"] Oct 31 00:44:05.204553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.204533 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.234597 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.234249 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:05.248689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.244939 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:05.358865 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.352350 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" event={"ID":"62b9146d-33e4-4a0b-a3ec-8c69f051e30d","Type":"ContainerStarted","Data":"56cb0391756cabdaec4aed1b2bd73c5022661c0fb5208763014992aab2d05549"} Oct 31 00:44:05.358865 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.352420 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" event={"ID":"62b9146d-33e4-4a0b-a3ec-8c69f051e30d","Type":"ContainerStarted","Data":"e398e60f37eab459cd90c59aab30950a4d61933ff2fb07d04e4fd28088e3a8d7"} Oct 31 00:44:05.425255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.402267 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6"] Oct 31 00:44:05.425255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.403436 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.425255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.403523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.425255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.403981 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdssg\" (UniqueName: \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.519341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.507912 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdssg\" (UniqueName: \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.519341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.508036 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.519341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.508100 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.519341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.509139 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.743890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.743858 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdssg\" (UniqueName: \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.762236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.760682 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podStartSLOduration=2.760662268 podStartE2EDuration="2.760662268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:05.575963282 +0000 UTC m=+1514.025050932" watchObservedRunningTime="2026-10-31 00:44:05.760662268 +0000 UTC m=+1514.209749908" Oct 31 00:44:05.770860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.768802 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e236238-7h4d6\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:05.921232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.920545 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:05.936102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.924323 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:05.936102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.924396 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:05.961230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:05.958452 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:06.268431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.267693 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:06.384262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.366538 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.517428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.517389 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.626345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.626299 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.646392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.646300 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.712289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.709033 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.712289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.711677 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:06.848265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.796803 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 31 00:44:06.848559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.847293 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.923302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.910718 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.088666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.971959 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.088666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.972536 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:07.088666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.972577 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:07.088666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:06.972967 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.169774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.167755 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:44:07.169774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.167860 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:44:07.445753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.442552 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.445753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.443196 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:44:07.548597 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.548542 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:44:07.579239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.577565 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 31 00:44:07.580396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.580275 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:44:07.637456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.637415 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.975232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.972638 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.991717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:07.991669 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:08.018568 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.018518 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:08.089236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.086324 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:08.089236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.086394 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:08.180765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.180721 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.209483 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.209443 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:44:08.221264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.179577 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:08.247420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.247312 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:08.251713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.251592 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:44:08.303485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.303446 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:44:08.467282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.405967 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.487018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.486965 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.557152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.557052 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.799932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.799893 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.984998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:08.984948 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.013721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.013685 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:09.166245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.159358 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:09.166245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.014505 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 00:44:09.166245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.014996 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.166245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.032353 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.014044 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.144312 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:09.040795 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.353s" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167090 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" event={"ID":"6baacca0-71e8-4389-a33e-a05602e81395","Type":"ContainerStarted","Data":"fd2109368a1ad0d4f672a9040f58d34c51f58dd432a1450bb3715eebeba334fb"} Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167226 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6"] Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167318 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167358 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167394 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167427 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167459 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167493 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167525 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167562 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167596 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:44:09.186707 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.167631 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:44:09.244041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.216297 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.244041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.234443 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:09.244041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.234810 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:44:09.264230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.262467 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:44:09.288230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.283307 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:09.364163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.362920 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.480231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.479332 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.510289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.480910 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:44:09.510289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.494444 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.562662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.562622 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.701376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.701330 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.818649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.818600 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.973999 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.958831 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:09.973999 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.959036 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:09.995483 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:09.995446 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:44:10.056629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.032856 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.161249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.161142 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.161552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.161528 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:10.183467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.183420 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:10.265398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.265353 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.290232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.266587 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.327308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.327266 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:10.353224 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.331676 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.389611 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.371289 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.389611 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.381563 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.481400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.481298 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:10.502085 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.502048 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.633646 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.633590 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.701387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.701343 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:10.827472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.827167 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:44:10.954018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.953983 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:10.954340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:10.954290 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:11.046508 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.046401 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.046935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.046904 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.206026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.205981 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.268810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.268318 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.294477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.294421 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.333363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.333320 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.412232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.384370 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 31 00:44:11.412232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.384725 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.412232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.385038 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:44:11.431232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.423717 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.573306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.573248 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.575358 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.575278 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:44:11.635699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.635661 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.755237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.739864 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:11.847558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:11.847511 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:44:12.140647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.140591 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:44:12.285529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.285489 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:12.285838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.285789 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.286850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.286813 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:12.287022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.287001 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.294171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.294137 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.325785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.324040 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:12.325785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.324118 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.365284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.350611 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:12.427383 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.427325 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.481232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.446335 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:12.595220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.499165 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:44:12.597015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.499237 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:44:12.623886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.531491 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:12.688395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.688352 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:44:12.690395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.690359 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.690709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.690686 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:12.775362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.775307 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.776921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.776898 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:12.894408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.894304 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:12.958260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.938043 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:12.958260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.938102 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:12.958260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.938465 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:12.958260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:12.938510 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:13.428346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.428295 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.490931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.490476 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:44:13.493271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.493245 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:44:13.521798 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.521767 5011 generic.go:356] "Generic (PLEG): container finished" podID="6baacca0-71e8-4389-a33e-a05602e81395" containerID="428fe6728c19102c74d63b3218c23fbf40a805e26fdf054817c3b89f836d26a1" exitCode=0 Oct 31 00:44:13.522128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.522105 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" event={"ID":"6baacca0-71e8-4389-a33e-a05602e81395","Type":"ContainerDied","Data":"428fe6728c19102c74d63b3218c23fbf40a805e26fdf054817c3b89f836d26a1"} Oct 31 00:44:13.611230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.610378 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:44:13.659815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.640294 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.659815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.640487 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.659815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.657192 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:13.672387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.672354 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.693874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.692974 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.706152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.695445 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:13.714697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.714665 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.715322 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.715291 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:13.715710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.715687 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:44:13.782233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.782059 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:44:13.782233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.782122 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:44:13.796661 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.796629 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:44:13.876381 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.871954 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:44:13.891899 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:13.891858 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baacca0_71e8_4389_a33e_a05602e81395.slice/crio-conmon-428fe6728c19102c74d63b3218c23fbf40a805e26fdf054817c3b89f836d26a1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baacca0_71e8_4389_a33e_a05602e81395.slice/crio-428fe6728c19102c74d63b3218c23fbf40a805e26fdf054817c3b89f836d26a1.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:44:13.898689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.898121 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:44:13.898689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.898292 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:44:13.898689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.898387 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:44:13.898689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.898539 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:44:13.904449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.904427 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:44:13.925329 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.925300 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:13.925559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:13.925534 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:14.870749 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.870671 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:14.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.922798 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\") pod \"6baacca0-71e8-4389-a33e-a05602e81395\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " Oct 31 00:44:14.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.922906 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\") pod \"6baacca0-71e8-4389-a33e-a05602e81395\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " Oct 31 00:44:14.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.922941 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\" (UniqueName: \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\") pod \"6baacca0-71e8-4389-a33e-a05602e81395\" (UID: \"6baacca0-71e8-4389-a33e-a05602e81395\") " Oct 31 00:44:14.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.927535 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg" pod "6baacca0-71e8-4389-a33e-a05602e81395" (UID: "6baacca0-71e8-4389-a33e-a05602e81395"). InnerVolumeSpecName "kube-api-access-xdssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:14.944261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.942595 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:14.944261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.942660 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:14.953990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.953959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca" pod "6baacca0-71e8-4389-a33e-a05602e81395" (UID: "6baacca0-71e8-4389-a33e-a05602e81395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:14.957097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:14.957071 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca" pod "6baacca0-71e8-4389-a33e-a05602e81395" (UID: "6baacca0-71e8-4389-a33e-a05602e81395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:15.040863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.040828 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:15.041056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.041040 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baacca0-71e8-4389-a33e-a05602e81395-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:15.041172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.041158 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdssg\" (UniqueName: \"kubernetes.io/projected/6baacca0-71e8-4389-a33e-a05602e81395-kube-api-access-xdssg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:15.612059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.612019 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" event={"ID":"6baacca0-71e8-4389-a33e-a05602e81395","Type":"ContainerDied","Data":"fd2109368a1ad0d4f672a9040f58d34c51f58dd432a1450bb3715eebeba334fb"} Oct 31 00:44:15.612289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.612086 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2109368a1ad0d4f672a9040f58d34c51f58dd432a1450bb3715eebeba334fb" Oct 31 00:44:15.612289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.612158 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6" Oct 31 00:44:15.761249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.761157 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7"] Oct 31 00:44:15.762058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.762034 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6baacca0-71e8-4389-a33e-a05602e81395" containerName="revoke-keys" Oct 31 00:44:15.762058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.762057 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6baacca0-71e8-4389-a33e-a05602e81395" containerName="revoke-keys" Oct 31 00:44:15.762306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.762288 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6baacca0-71e8-4389-a33e-a05602e81395" containerName="revoke-keys" Oct 31 00:44:15.770727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.766460 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.781700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.781275 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:15.781700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.781412 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:15.785930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.783275 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7"] Oct 31 00:44:15.871308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.869281 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.871308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.869602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkxh\" (UniqueName: \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.871308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.869666 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.955259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.951537 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:15.955259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.951599 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:15.970903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.970869 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.971064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.971051 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkxh\" (UniqueName: \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.971141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.971086 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.971563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.971537 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.973699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.973663 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:15.993085 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:15.993053 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkxh\" (UniqueName: \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\") pod \"maas-api-revoke-sub-61925b28-dwlg7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:16.085802 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:16.085775 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:16.432061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:16.432024 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7"] Oct 31 00:44:16.623877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:16.623836 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" event={"ID":"4efe2d40-b8bf-4089-8c4c-4671a008eaf7","Type":"ContainerStarted","Data":"e790c474d8369d19efa5f58f5d5e55da2dc1edff2d189f5c2e6b59fda30b70a3"} Oct 31 00:44:16.920656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:16.920616 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:44:16.921148 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:16.920679 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:44:17.654544 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:17.654345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" event={"ID":"4efe2d40-b8bf-4089-8c4c-4671a008eaf7","Type":"ContainerStarted","Data":"475bc9f04e487980431cf392e160fed7ac38ba107efff72a51edc9c503710123"} Oct 31 00:44:18.671557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.671514 5011 generic.go:356] "Generic (PLEG): container finished" podID="4efe2d40-b8bf-4089-8c4c-4671a008eaf7" containerID="475bc9f04e487980431cf392e160fed7ac38ba107efff72a51edc9c503710123" exitCode=0 Oct 31 00:44:18.672064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.671765 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" event={"ID":"4efe2d40-b8bf-4089-8c4c-4671a008eaf7","Type":"ContainerDied","Data":"475bc9f04e487980431cf392e160fed7ac38ba107efff72a51edc9c503710123"} Oct 31 00:44:18.906632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.906607 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:18.918104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.918081 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:18.959423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.959342 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:18.959700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:18.959657 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:19.007618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.007584 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:19.035289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.035254 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\") pod \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " Oct 31 00:44:19.035585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.035569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\" (UniqueName: \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\") pod \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " Oct 31 00:44:19.035676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.035593 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca" pod "4efe2d40-b8bf-4089-8c4c-4671a008eaf7" (UID: "4efe2d40-b8bf-4089-8c4c-4671a008eaf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:19.035676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.035631 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\") pod \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\" (UID: \"4efe2d40-b8bf-4089-8c4c-4671a008eaf7\") " Oct 31 00:44:19.036368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.036343 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:19.038051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.038016 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca" pod "4efe2d40-b8bf-4089-8c4c-4671a008eaf7" (UID: "4efe2d40-b8bf-4089-8c4c-4671a008eaf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:19.038485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.038461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh" pod "4efe2d40-b8bf-4089-8c4c-4671a008eaf7" (UID: "4efe2d40-b8bf-4089-8c4c-4671a008eaf7"). InnerVolumeSpecName "kube-api-access-fqkxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:19.138319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.138283 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkxh\" (UniqueName: \"kubernetes.io/projected/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-kube-api-access-fqkxh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:19.138319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.138314 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2d40-b8bf-4089-8c4c-4671a008eaf7-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:19.682268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.682241 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" Oct 31 00:44:19.685649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.685621 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7" event={"ID":"4efe2d40-b8bf-4089-8c4c-4671a008eaf7","Type":"ContainerDied","Data":"e790c474d8369d19efa5f58f5d5e55da2dc1edff2d189f5c2e6b59fda30b70a3"} Oct 31 00:44:19.685774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.685667 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e790c474d8369d19efa5f58f5d5e55da2dc1edff2d189f5c2e6b59fda30b70a3" Oct 31 00:44:19.685774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:19.685721 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:44:20.387095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.387061 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4"] Oct 31 00:44:20.387718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.387704 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4efe2d40-b8bf-4089-8c4c-4671a008eaf7" containerName="revoke-keys" Oct 31 00:44:20.387782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.387720 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe2d40-b8bf-4089-8c4c-4671a008eaf7" containerName="revoke-keys" Oct 31 00:44:20.387868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.387859 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe2d40-b8bf-4089-8c4c-4671a008eaf7" containerName="revoke-keys" Oct 31 00:44:20.391635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.391618 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.420999 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.420963 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:20.422475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.422446 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:20.423095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.423065 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4"] Oct 31 00:44:20.487375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.487343 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:44:20.492010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.491990 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.508624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.508600 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d66e61c1\"" Oct 31 00:44:20.508779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.508706 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d66e61c1-dockercfg-k8tgl\"" Oct 31 00:44:20.510924 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.510896 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:44:20.568020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.567126 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpz8\" (UniqueName: \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.568020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.567349 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.568020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.567902 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.674614 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.674839 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.674870 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4w5\" (UniqueName: \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.674972 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.675035 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpz8\" (UniqueName: \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.681567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.676276 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.710991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.707761 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.725302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.725270 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpz8\" (UniqueName: \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\") pod \"maas-api-revoke-sub-d6c04e1f-zgqm4\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:20.777138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.777094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4w5\" (UniqueName: \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.777323 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.777237 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.778009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.777992 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.801394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.801355 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4w5\" (UniqueName: \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\") pod \"payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:20.807158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:20.807130 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:21.008764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.008683 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:21.142538 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:21.140564 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6792620a_d102_4a41_be55_52e46d12adf1.slice/crio-d05d77845540693f43b24cb69d5e9d051e14a34c106248f80423b0de984c63c1 WatchSource:0}: Error finding container d05d77845540693f43b24cb69d5e9d051e14a34c106248f80423b0de984c63c1: Status 404 returned error can't find the container with id d05d77845540693f43b24cb69d5e9d051e14a34c106248f80423b0de984c63c1 Oct 31 00:44:21.142538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.140959 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:44:21.335698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.335237 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4"] Oct 31 00:44:21.729932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.729887 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" event={"ID":"6792620a-d102-4a41-be55-52e46d12adf1","Type":"ContainerStarted","Data":"d05d77845540693f43b24cb69d5e9d051e14a34c106248f80423b0de984c63c1"} Oct 31 00:44:21.732409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.732381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" event={"ID":"1eb4d46d-477f-4bc3-975d-edefb7cb6249","Type":"ContainerStarted","Data":"2d155a06ca4ed97d0aa323cd00d50b4aa5cdd16276207866169662cfb8fbee8c"} Oct 31 00:44:21.732551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.732426 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" event={"ID":"1eb4d46d-477f-4bc3-975d-edefb7cb6249","Type":"ContainerStarted","Data":"d60ba0e1dd33dad077ebf99743fa38dea54abee843c6a2dac93deac5639393c5"} Oct 31 00:44:21.784727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:21.777082 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" podStartSLOduration=1.777062871 podStartE2EDuration="1.777062871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:21.771817155 +0000 UTC m=+1530.220904794" watchObservedRunningTime="2026-10-31 00:44:21.777062871 +0000 UTC m=+1530.226150510" Oct 31 00:44:22.758664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:22.758579 5011 generic.go:356] "Generic (PLEG): container finished" podID="1eb4d46d-477f-4bc3-975d-edefb7cb6249" containerID="2d155a06ca4ed97d0aa323cd00d50b4aa5cdd16276207866169662cfb8fbee8c" exitCode=0 Oct 31 00:44:22.759092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:22.758776 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" event={"ID":"1eb4d46d-477f-4bc3-975d-edefb7cb6249","Type":"ContainerDied","Data":"2d155a06ca4ed97d0aa323cd00d50b4aa5cdd16276207866169662cfb8fbee8c"} Oct 31 00:44:23.769839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:23.769805 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" event={"ID":"6792620a-d102-4a41-be55-52e46d12adf1","Type":"ContainerStarted","Data":"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68"} Oct 31 00:44:23.770269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:23.769911 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:23.812399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:23.812294 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" podStartSLOduration=1.402362121 podStartE2EDuration="3.812277483s" totalImagesPullingTime="2.409915362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:23.809327235 +0000 UTC m=+1532.258414881" watchObservedRunningTime="2026-10-31 00:44:23.812277483 +0000 UTC m=+1532.261365124" Oct 31 00:44:23.988229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:23.986055 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:24.037365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.037328 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\") pod \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " Oct 31 00:44:24.037586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.037566 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\" (UniqueName: \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\") pod \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " Oct 31 00:44:24.037741 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.037722 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\") pod \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\" (UID: \"1eb4d46d-477f-4bc3-975d-edefb7cb6249\") " Oct 31 00:44:24.039260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.038953 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca" pod "1eb4d46d-477f-4bc3-975d-edefb7cb6249" (UID: "1eb4d46d-477f-4bc3-975d-edefb7cb6249"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:24.042602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.042566 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca" pod "1eb4d46d-477f-4bc3-975d-edefb7cb6249" (UID: "1eb4d46d-477f-4bc3-975d-edefb7cb6249"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:24.060396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.044412 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8" pod "1eb4d46d-477f-4bc3-975d-edefb7cb6249" (UID: "1eb4d46d-477f-4bc3-975d-edefb7cb6249"). InnerVolumeSpecName "kube-api-access-sfpz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:24.139826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.139783 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpz8\" (UniqueName: \"kubernetes.io/projected/1eb4d46d-477f-4bc3-975d-edefb7cb6249-kube-api-access-sfpz8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:24.139826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.139824 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:24.140033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.139839 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb4d46d-477f-4bc3-975d-edefb7cb6249-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:24.222563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.222530 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:44:24.224017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.223991 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1eb4d46d-477f-4bc3-975d-edefb7cb6249" containerName="revoke-keys" Oct 31 00:44:24.224017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.224016 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb4d46d-477f-4bc3-975d-edefb7cb6249" containerName="revoke-keys" Oct 31 00:44:24.224257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.224240 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb4d46d-477f-4bc3-975d-edefb7cb6249" containerName="revoke-keys" Oct 31 00:44:24.229038 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.229002 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.258126 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.258090 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:44:24.262604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.259382 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a9461dc0-openshift-default-dockercfg-5lz78\"" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361197 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk8k\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361412 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361462 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361663 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361712 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361754 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.363231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.361808 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.463862 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.463943 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.463994 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464019 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464156 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk8k\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464264 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.464767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464373 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.470554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.464907 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.470554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.465913 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.470554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.466148 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.470554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.466819 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.470554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.467009 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.471764 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.471710 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.473460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.472785 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.526376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.523797 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.527317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.527247 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk8k\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\") pod \"e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.574325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.574287 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:24.816296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.815443 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" Oct 31 00:44:24.840720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.838638 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4" event={"ID":"1eb4d46d-477f-4bc3-975d-edefb7cb6249","Type":"ContainerDied","Data":"d60ba0e1dd33dad077ebf99743fa38dea54abee843c6a2dac93deac5639393c5"} Oct 31 00:44:24.840720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.838764 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60ba0e1dd33dad077ebf99743fa38dea54abee843c6a2dac93deac5639393c5" Oct 31 00:44:24.938861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.938830 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:44:24.939069 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:24.939047 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267960c0_24ec_422e_bde3_4068242dff0c.slice/crio-1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7 WatchSource:0}: Error finding container 1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7: Status 404 returned error can't find the container with id 1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7 Oct 31 00:44:24.942723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.942681 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:24.942812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.942772 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:24.942863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.942818 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:24.984426 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.984391 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7"] Oct 31 00:44:24.990696 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:24.990671 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.005656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.004593 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:25.005656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.004770 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:25.011474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.011449 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7"] Oct 31 00:44:25.081167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.081123 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.081362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.081225 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.081425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.081362 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.183529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.183492 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.183721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.183567 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.183721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.183707 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.184727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.184305 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.188385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.188358 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.209489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.209462 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") pod \"maas-api-revoke-sub-874ab0e4-chdz7\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.324928 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.324889 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:25.655307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.653638 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7"] Oct 31 00:44:25.820115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.820077 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" event={"ID":"5fde329c-96d4-4e48-bc77-97125cfa5ae2","Type":"ContainerStarted","Data":"3efd40a91068679b3ef37c77c8f2a5306e5c7038ce075a0105f9714bf2d524de"} Oct 31 00:44:25.822477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.822449 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerStarted","Data":"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351"} Oct 31 00:44:25.822615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.822510 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerStarted","Data":"1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7"} Oct 31 00:44:25.867572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:25.867517 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podStartSLOduration=1.867497384 podStartE2EDuration="1.867497384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:25.864793134 +0000 UTC m=+1534.313880773" watchObservedRunningTime="2026-10-31 00:44:25.867497384 +0000 UTC m=+1534.316585024" Oct 31 00:44:26.575262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.575232 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:26.582371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.582345 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:26.833509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.833475 5011 generic.go:356] "Generic (PLEG): container finished" podID="5fde329c-96d4-4e48-bc77-97125cfa5ae2" containerID="da42ce9bd839a4d1da2595f30c3a18533dbcced9642cfb813db591b104ebc20e" exitCode=0 Oct 31 00:44:26.833900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.833580 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" event={"ID":"5fde329c-96d4-4e48-bc77-97125cfa5ae2","Type":"ContainerDied","Data":"da42ce9bd839a4d1da2595f30c3a18533dbcced9642cfb813db591b104ebc20e"} Oct 31 00:44:26.834097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.834074 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:26.835318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:26.835291 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:44:28.181857 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.181831 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:28.280345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.278794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\") pod \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " Oct 31 00:44:28.280345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.278886 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") pod \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " Oct 31 00:44:28.280345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.278970 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\") pod \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " Oct 31 00:44:28.342239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.335875 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca" pod "5fde329c-96d4-4e48-bc77-97125cfa5ae2" (UID: "5fde329c-96d4-4e48-bc77-97125cfa5ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.392910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq" pod "5fde329c-96d4-4e48-bc77-97125cfa5ae2" (UID: "5fde329c-96d4-4e48-bc77-97125cfa5ae2"). InnerVolumeSpecName "kube-api-access-5lxxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.393357 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") pod \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\" (UID: \"5fde329c-96d4-4e48-bc77-97125cfa5ae2\") " Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.394013 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:28.394104 5011 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/5fde329c-96d4-4e48-bc77-97125cfa5ae2/volumes/kubernetes.io~projected/kube-api-access-5lxxq Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.394142 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq" pod "5fde329c-96d4-4e48-bc77-97125cfa5ae2" (UID: "5fde329c-96d4-4e48-bc77-97125cfa5ae2"). InnerVolumeSpecName "kube-api-access-5lxxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:28.403286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.396417 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca" pod "5fde329c-96d4-4e48-bc77-97125cfa5ae2" (UID: "5fde329c-96d4-4e48-bc77-97125cfa5ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:28.495448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.495412 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxxq\" (UniqueName: \"kubernetes.io/projected/5fde329c-96d4-4e48-bc77-97125cfa5ae2-kube-api-access-5lxxq\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:28.495448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.495450 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fde329c-96d4-4e48-bc77-97125cfa5ae2-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:28.958348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.958319 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" Oct 31 00:44:28.958541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.958359 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7" event={"ID":"5fde329c-96d4-4e48-bc77-97125cfa5ae2","Type":"ContainerDied","Data":"3efd40a91068679b3ef37c77c8f2a5306e5c7038ce075a0105f9714bf2d524de"} Oct 31 00:44:28.958541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:28.958428 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3efd40a91068679b3ef37c77c8f2a5306e5c7038ce075a0105f9714bf2d524de" Oct 31 00:44:29.747318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.747282 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:44:29.748140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.748116 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5fde329c-96d4-4e48-bc77-97125cfa5ae2" containerName="revoke-keys" Oct 31 00:44:29.748140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.748138 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fde329c-96d4-4e48-bc77-97125cfa5ae2" containerName="revoke-keys" Oct 31 00:44:29.748386 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.748371 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fde329c-96d4-4e48-bc77-97125cfa5ae2" containerName="revoke-keys" Oct 31 00:44:29.752718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.752696 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:29.774830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.774799 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:44:29.778386 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.778360 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a9461dc0\"" Oct 31 00:44:29.778489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.778438 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a9461dc0-dockercfg-7s76n\"" Oct 31 00:44:29.929229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.928602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:29.929229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:29.928758 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzfj\" (UniqueName: \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.030782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.030680 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.030964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.030827 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzfj\" (UniqueName: \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.031953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.031922 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.059455 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.055189 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzfj\" (UniqueName: \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\") pod \"payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.072114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.072065 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.398071 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.398038 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:44:30.399018 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:30.398985 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf643fb1a_8434_4331_94ed_f39dceddd005.slice/crio-5a4b8d43f0ae5c13c1a7ee37b4e94558d215520bf4c015febeb5d81a282e0ff5 WatchSource:0}: Error finding container 5a4b8d43f0ae5c13c1a7ee37b4e94558d215520bf4c015febeb5d81a282e0ff5: Status 404 returned error can't find the container with id 5a4b8d43f0ae5c13c1a7ee37b4e94558d215520bf4c015febeb5d81a282e0ff5 Oct 31 00:44:30.772025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.771944 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl"] Oct 31 00:44:30.778610 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.778582 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.801288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.801169 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl"] Oct 31 00:44:30.809232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.806293 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:30.809232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.806389 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:30.849086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.848360 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.849086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.848555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bpf\" (UniqueName: \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.849086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.848589 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.949723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.949687 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bpf\" (UniqueName: \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.949723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.949728 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.949975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.949890 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.950721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.950688 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.952384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.952360 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.971676 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.971655 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bpf\" (UniqueName: \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\") pod \"maas-api-revoke-sub-5792bd42-llqbl\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:30.992275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.992241 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" event={"ID":"f643fb1a-8434-4331-94ed-f39dceddd005","Type":"ContainerStarted","Data":"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab"} Oct 31 00:44:30.992501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.992482 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:30.992572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:30.992532 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" event={"ID":"f643fb1a-8434-4331-94ed-f39dceddd005","Type":"ContainerStarted","Data":"5a4b8d43f0ae5c13c1a7ee37b4e94558d215520bf4c015febeb5d81a282e0ff5"} Oct 31 00:44:31.037185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:31.037114 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" podStartSLOduration=2.037102488 podStartE2EDuration="2.037102488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:31.034556406 +0000 UTC m=+1539.483644046" watchObservedRunningTime="2026-10-31 00:44:31.037102488 +0000 UTC m=+1539.486190128" Oct 31 00:44:31.118371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:31.118344 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:31.447749 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:31.447697 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6965bd9b_801e_43f0_8084_952e13e2df39.slice/crio-c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc WatchSource:0}: Error finding container c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc: Status 404 returned error can't find the container with id c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc Oct 31 00:44:31.447892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:31.447828 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl"] Oct 31 00:44:31.951374 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:31.951332 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:44:32.003127 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:32.003091 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" event={"ID":"6965bd9b-801e-43f0-8084-952e13e2df39","Type":"ContainerStarted","Data":"30f405b6b82e51d75f5c6904c5cb2482b581692fd3033fc3c3623ec1d1bc3c77"} Oct 31 00:44:32.003301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:32.003160 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" event={"ID":"6965bd9b-801e-43f0-8084-952e13e2df39","Type":"ContainerStarted","Data":"c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc"} Oct 31 00:44:32.045014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:32.044955 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" podStartSLOduration=2.044934823 podStartE2EDuration="2.044934823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:32.040957828 +0000 UTC m=+1540.490045467" watchObservedRunningTime="2026-10-31 00:44:32.044934823 +0000 UTC m=+1540.494022462" Oct 31 00:44:33.012862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.012826 5011 generic.go:356] "Generic (PLEG): container finished" podID="6965bd9b-801e-43f0-8084-952e13e2df39" containerID="30f405b6b82e51d75f5c6904c5cb2482b581692fd3033fc3c3623ec1d1bc3c77" exitCode=0 Oct 31 00:44:33.013311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.012920 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" event={"ID":"6965bd9b-801e-43f0-8084-952e13e2df39","Type":"ContainerDied","Data":"30f405b6b82e51d75f5c6904c5cb2482b581692fd3033fc3c3623ec1d1bc3c77"} Oct 31 00:44:33.540252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.540217 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:44:33.546032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.546004 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.560454 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.559479 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f598cd05\"" Oct 31 00:44:33.563230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.563184 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:44:33.586452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.586427 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f598cd05-dockercfg-5fs8n\"" Oct 31 00:44:33.801605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.795820 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6jw\" (UniqueName: \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.801605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.796038 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.852824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.848982 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:44:33.855390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.855363 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:44:33.855516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.855466 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:33.901593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.901555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:33.901772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.901705 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65jn\" (UniqueName: \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:33.901888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.901871 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6jw\" (UniqueName: \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.902135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.902118 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.904556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.904525 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:33.931290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:33.931253 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6jw\" (UniqueName: \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\") pod \"payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:34.004116 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.004071 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:34.004308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.004130 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65jn\" (UniqueName: \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:34.004943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.004920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:34.027629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.027409 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65jn\" (UniqueName: \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\") pod \"payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:34.164475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.164437 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:34.173262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.173081 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:34.257339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.257228 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:34.314247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.313801 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\" (UniqueName: \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\") pod \"6965bd9b-801e-43f0-8084-952e13e2df39\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " Oct 31 00:44:34.314247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.314053 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\") pod \"6965bd9b-801e-43f0-8084-952e13e2df39\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " Oct 31 00:44:34.314247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.314126 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\") pod \"6965bd9b-801e-43f0-8084-952e13e2df39\" (UID: \"6965bd9b-801e-43f0-8084-952e13e2df39\") " Oct 31 00:44:34.315829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.315579 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca" pod "6965bd9b-801e-43f0-8084-952e13e2df39" (UID: "6965bd9b-801e-43f0-8084-952e13e2df39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:34.321385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.321345 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf" pod "6965bd9b-801e-43f0-8084-952e13e2df39" (UID: "6965bd9b-801e-43f0-8084-952e13e2df39"). InnerVolumeSpecName "kube-api-access-f6bpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:34.321607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.321573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca" pod "6965bd9b-801e-43f0-8084-952e13e2df39" (UID: "6965bd9b-801e-43f0-8084-952e13e2df39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:34.417325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.417251 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:34.417325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.417279 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6965bd9b-801e-43f0-8084-952e13e2df39-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:34.417325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.417293 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bpf\" (UniqueName: \"kubernetes.io/projected/6965bd9b-801e-43f0-8084-952e13e2df39-kube-api-access-f6bpf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:44:34.520572 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:34.520538 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987e4abc_a0c9_47f2_b3a5_b291b358cede.slice/crio-b8f7f6af8d74c8a9cf592e5f89d2480ac198b87ed5829fe7fa96e1f28597bbd9 WatchSource:0}: Error finding container b8f7f6af8d74c8a9cf592e5f89d2480ac198b87ed5829fe7fa96e1f28597bbd9: Status 404 returned error can't find the container with id b8f7f6af8d74c8a9cf592e5f89d2480ac198b87ed5829fe7fa96e1f28597bbd9 Oct 31 00:44:34.521672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.521487 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:44:34.534399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.534372 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:44:34.537175 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:34.537152 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6c69c0_0b49_46e8_9573_5eac80d0f7a0.slice/crio-171f4db493ba482b00f56b3b40fcaaeee60ed0c47b6f109f619f366b11acbbbc WatchSource:0}: Error finding container 171f4db493ba482b00f56b3b40fcaaeee60ed0c47b6f109f619f366b11acbbbc: Status 404 returned error can't find the container with id 171f4db493ba482b00f56b3b40fcaaeee60ed0c47b6f109f619f366b11acbbbc Oct 31 00:44:34.818775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:34.818738 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:44:35.038956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.038917 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" event={"ID":"987e4abc-a0c9-47f2-b3a5-b291b358cede","Type":"ContainerStarted","Data":"237eda4d4fba7a5cf8abfcffea03aa8ddd3e927dd1a0beb64021a7fc1e276dbc"} Oct 31 00:44:35.039437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.038988 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" event={"ID":"987e4abc-a0c9-47f2-b3a5-b291b358cede","Type":"ContainerStarted","Data":"b8f7f6af8d74c8a9cf592e5f89d2480ac198b87ed5829fe7fa96e1f28597bbd9"} Oct 31 00:44:35.039437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.039079 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:35.040976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.040948 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" event={"ID":"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0","Type":"ContainerStarted","Data":"9b39dac53c003be9ed4a2950c9e3b4670e687e68a2e6c6938bdd45d2c93321c7"} Oct 31 00:44:35.041114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.041021 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" event={"ID":"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0","Type":"ContainerStarted","Data":"171f4db493ba482b00f56b3b40fcaaeee60ed0c47b6f109f619f366b11acbbbc"} Oct 31 00:44:35.041380 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.041293 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:35.042662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.042640 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" event={"ID":"6965bd9b-801e-43f0-8084-952e13e2df39","Type":"ContainerDied","Data":"c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc"} Oct 31 00:44:35.042779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.042675 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c50804de19119f4acf4fe7473e00aafda4d742380be60b3397c86135d09526dc" Oct 31 00:44:35.042779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.042732 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl" Oct 31 00:44:35.080118 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.080046 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" podStartSLOduration=2.080034588 podStartE2EDuration="2.080034588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:35.076463697 +0000 UTC m=+1543.525551338" watchObservedRunningTime="2026-10-31 00:44:35.080034588 +0000 UTC m=+1543.529122280" Oct 31 00:44:35.130741 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:35.130698 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" podStartSLOduration=2.130685902 podStartE2EDuration="2.130685902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:35.127517609 +0000 UTC m=+1543.576605259" watchObservedRunningTime="2026-10-31 00:44:35.130685902 +0000 UTC m=+1543.579773542" Oct 31 00:44:36.344618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.344584 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:44:36.345501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.345477 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6965bd9b-801e-43f0-8084-952e13e2df39" containerName="revoke-keys" Oct 31 00:44:36.345501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.345499 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6965bd9b-801e-43f0-8084-952e13e2df39" containerName="revoke-keys" Oct 31 00:44:36.345721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.345702 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6965bd9b-801e-43f0-8084-952e13e2df39" containerName="revoke-keys" Oct 31 00:44:36.365889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.365859 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.375746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.375716 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:44:36.379852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.379701 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d66e61c1-serving-cert\"" Oct 31 00:44:36.447135 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.447105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.447385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.447364 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjpx\" (UniqueName: \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.447475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.447426 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.447532 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.447486 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.549046 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:36.549191 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-serving-cert: secret "maas-api-e2e-ait-d66e61c1-serving-cert" not found Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:36.549284 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls podName:243abd72-5ae5-4b9d-a741-ec02d42b89db nodeName:}" failed. No retries permitted until 2026-10-31 00:44:37.049260753 +0000 UTC m=+1545.498348387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls") pod "maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" (UID: "243abd72-5ae5-4b9d-a741-ec02d42b89db") : secret "maas-api-e2e-ait-d66e61c1-serving-cert" not found Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.549432 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjpx\" (UniqueName: \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.549505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.549554 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.556509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.553245 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.560078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.559621 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:36.581375 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:36.581326 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjpx\" (UniqueName: \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:37.056852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:37.056817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:37.059835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:37.059809 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") pod \"maas-api-e2e-ait-d66e61c1-755b844887-tjsp9\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:37.289327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:37.289292 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:37.664244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:37.661230 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:44:37.670345 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:37.670311 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243abd72_5ae5_4b9d_a741_ec02d42b89db.slice/crio-1639cbeedf3f2f3878cba18a272b18e8cf818a67cf2ce3f72ccf082864870923 WatchSource:0}: Error finding container 1639cbeedf3f2f3878cba18a272b18e8cf818a67cf2ce3f72ccf082864870923: Status 404 returned error can't find the container with id 1639cbeedf3f2f3878cba18a272b18e8cf818a67cf2ce3f72ccf082864870923 Oct 31 00:44:38.081656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:38.081616 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" event={"ID":"243abd72-5ae5-4b9d-a741-ec02d42b89db","Type":"ContainerStarted","Data":"1639cbeedf3f2f3878cba18a272b18e8cf818a67cf2ce3f72ccf082864870923"} Oct 31 00:44:39.095117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:39.095080 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" event={"ID":"243abd72-5ae5-4b9d-a741-ec02d42b89db","Type":"ContainerStarted","Data":"26f518f42d939dcc8b9bb8230449ef50afc764745adb12225fc18440f4103b16"} Oct 31 00:44:39.095588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:39.095216 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:39.137493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:39.137446 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" podStartSLOduration=2.808252222 podStartE2EDuration="3.137431957s" totalImagesPullingTime="329.179735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:39.134983447 +0000 UTC m=+1547.584071086" watchObservedRunningTime="2026-10-31 00:44:39.137431957 +0000 UTC m=+1547.586519596" Oct 31 00:44:42.019230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:42.009294 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:44:42.332401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:42.326802 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:42.332401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:42.326899 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:42.588307 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:42.567282 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:42.719959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:42.719320 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.201231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.200606 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.267232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.260889 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.310369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.310325 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.365336 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.365292 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.387397 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.387354 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.507839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.507740 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.508665 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.508054 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:43.618493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.618449 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:44:43.620964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.620937 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:44:43.650369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.650322 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.694673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.694557 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.723633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.723162 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.871385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:43.871341 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.221980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.220801 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:44.278232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.261308 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.327233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.281920 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:44.327233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.281990 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:44:44.327233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.282038 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:44:44.406338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.398373 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.421231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.410397 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.421231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.419727 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:44:44.421231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.419798 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:44:44.421720 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.421682 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:44:44.452162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.422385 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:44:44.452162 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.423433 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.598265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.598131 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.631463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.631421 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:44:44.632879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.632848 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.697236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.679363 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.727596 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.701237 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.738266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.728341 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:44:44.738266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.728417 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:44:44.738266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.727106 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:44.850382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.850278 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 00:44:44.894234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.874395 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.894234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:44.884359 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:44:45.075196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.075151 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.188288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.181122 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:45.188288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.181201 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.626236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.620442 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.697709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.637780 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:45.713014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.708962 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:44:45.713014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.710265 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.713014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.668437 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.741234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.638052 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:44:45.743335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.743300 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:45.743539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.743516 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.743659 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.668104 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:44:45.744850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.740032 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.841233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.748582 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:44:46.052288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.748856 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:46.052288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.785660 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:46.052288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:45.984050 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:44:46.187886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.187840 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.713232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.709188 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.775369 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.775309 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.815231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.800313 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:46.815231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.800358 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:46.815231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.809304 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:44:46.822339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.800378 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:44:46.894462 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:46.826685 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.127s" Oct 31 00:44:46.894854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.894833 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:44:46.895034 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.895022 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:44:46.895164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.895153 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:44:46.895321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.895309 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:44:46.895469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.895457 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:44:46.896507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.834275 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:44:46.915830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.915791 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 31 00:44:46.996295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.996014 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.002979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.998045 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:47.002979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.998793 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:44:47.002979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.999378 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:44:47.002979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:46.999922 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:47.196231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.194233 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:44:47.461404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.461373 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:44:47.465873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.465848 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:44:47.566232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.560971 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:44:47.566232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.561314 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:44:47.566232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.561367 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:44:47.566232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.561402 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:44:47.868294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.719154 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:44:47.921789 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:47.912395 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:44:48.081062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.036634 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.153296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.111967 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.231823 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.231780 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.232978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.231782 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.508288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.475238 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:44:48.543231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.525287 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:44:48.543231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.526022 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:44:48.775424 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.775327 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.913463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.913409 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.915474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.915401 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.925664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.925635 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.962049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:48.961340 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.007102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.006050 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.041277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.037304 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.041277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.038577 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.128717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.128675 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:44:49.135255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.131300 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:44:49.184382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.184336 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.271514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.270502 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.275505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.270539 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.482972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.482939 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:44:49.491512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.491482 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:44:49.494556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.494525 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:44:49.522425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.522400 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:44:49.524202 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:49.524183 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:44:54.052141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.052101 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf"] Oct 31 00:44:54.062365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.062334 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a7b93c-dzbwf"] Oct 31 00:44:54.656658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.656626 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:44:54.668292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.667391 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.683156 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.683096 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a9461dc0-serving-cert\"" Oct 31 00:44:54.692132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.692105 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:44:54.837376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.837313 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.837376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.837370 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.837716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.837452 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h267r\" (UniqueName: \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.837716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.837527 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.939714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.939619 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h267r\" (UniqueName: \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.939714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.939678 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.939958 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:54.939853 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-serving-cert: secret "maas-api-e2e-ait-a9461dc0-serving-cert" not found Oct 31 00:44:54.939958 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:44:54.939929 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls podName:56306f5e-9d2b-45be-bfb6-58804eb50c08 nodeName:}" failed. No retries permitted until 2026-10-31 00:44:55.43990871 +0000 UTC m=+1563.888996328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls") pod "maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" (UID: "56306f5e-9d2b-45be-bfb6-58804eb50c08") : secret "maas-api-e2e-ait-a9461dc0-serving-cert" not found Oct 31 00:44:54.940078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.940052 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.940133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.940092 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.940827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.940806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.942728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.942705 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:54.967961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:54.967932 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h267r\" (UniqueName: \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:55.120681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.120648 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 00:44:55.125653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.125630 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.140632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.140561 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 00:44:55.140805 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.140783 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zgxxb\"" Oct 31 00:44:55.140920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.140624 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:44:55.140986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.140655 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:44:55.140986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.140584 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:44:55.246105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246024 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w6f\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246160 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246322 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246383 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246538 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.246692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.246647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348572 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348672 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348793 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.348960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348836 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.349071 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.348984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w6f\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.349791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.349764 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.351325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.351298 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.351815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.351567 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.351815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.351712 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.351815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.351774 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.370257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.370232 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.370625 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.370605 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w6f\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s7c69\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.443637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.443610 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:55.452613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.452584 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:55.455679 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.455649 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") pod \"maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:55.601856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.601814 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:55.732238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.728182 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b403f37-1e3f-4bd1-a369-56d7ec847455" path="/var/lib/kubelet/pods/3b403f37-1e3f-4bd1-a369-56d7ec847455/volumes" Oct 31 00:44:55.872687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.872654 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 00:44:55.875388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.873233 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:44:55.976825 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:44:55.976766 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56306f5e_9d2b_45be_bfb6_58804eb50c08.slice/crio-e5dcc998f10edff995cfabfe7befc6c920e43f03172505918c13da971417faca WatchSource:0}: Error finding container e5dcc998f10edff995cfabfe7befc6c920e43f03172505918c13da971417faca: Status 404 returned error can't find the container with id e5dcc998f10edff995cfabfe7befc6c920e43f03172505918c13da971417faca Oct 31 00:44:55.980510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:55.980480 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:44:56.747984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:56.747941 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" event={"ID":"3caa6982-ed49-43b3-b239-7d6c98ca7abd","Type":"ContainerStarted","Data":"9d51dec6ff3be5c80b327cf3da5f9679b13187f64b5112d92d8020f6dba48843"} Oct 31 00:44:56.750829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:56.750794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" event={"ID":"56306f5e-9d2b-45be-bfb6-58804eb50c08","Type":"ContainerStarted","Data":"24d3787cc847edb01420bc6ea7eaf340b0f7d8728d9aceb476685cfdbc91adab"} Oct 31 00:44:56.750981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:56.750846 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" event={"ID":"56306f5e-9d2b-45be-bfb6-58804eb50c08","Type":"ContainerStarted","Data":"e5dcc998f10edff995cfabfe7befc6c920e43f03172505918c13da971417faca"} Oct 31 00:44:56.750981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:56.750957 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:44:56.802472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:56.802418 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" podStartSLOduration=2.488851813 podStartE2EDuration="2.802403686s" totalImagesPullingTime="313.551873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:56.798615984 +0000 UTC m=+1565.247703624" watchObservedRunningTime="2026-10-31 00:44:56.802403686 +0000 UTC m=+1565.251491324" Oct 31 00:44:58.486200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:58.486157 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:58.486644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:58.486261 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:58.771322 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:58.771227 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" event={"ID":"3caa6982-ed49-43b3-b239-7d6c98ca7abd","Type":"ContainerStarted","Data":"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593"} Oct 31 00:44:58.771491 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:58.771355 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:44:58.835561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:58.835510 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" podStartSLOduration=1.224611315 podStartE2EDuration="3.83549383s" totalImagesPullingTime="2.610882515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:58.821153201 +0000 UTC m=+1567.270240839" watchObservedRunningTime="2026-10-31 00:44:58.83549383 +0000 UTC m=+1567.284581470" Oct 31 00:44:59.782402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:59.782368 5011 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s7c69 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:44:59.782890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:44:59.782419 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:00.189028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.188994 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:00.194420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.194397 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.198035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.198009 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:00.203671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.203644 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.244025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.243995 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:00.249316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.249283 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:00.253331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.253310 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:00.257909 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.257890 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:00.258077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.258045 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.265582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.265554 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:45:00.277467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.268582 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.296794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.296770 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:00.307286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.306311 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.307286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.306381 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbmz\" (UniqueName: \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.307286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.306489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jp72\" (UniqueName: \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.307286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.306802 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.309294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.309119 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:00.408386 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408347 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jp72\" (UniqueName: \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.408576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408401 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6h4\" (UniqueName: \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.408576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408495 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574ng\" (UniqueName: \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.408729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408704 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.408868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408846 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.408976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.408960 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.409059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.409045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.409115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.409101 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbmz\" (UniqueName: \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.409762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.409737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.409854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.409838 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.430730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.430706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jp72\" (UniqueName: \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.430889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.430773 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbmz\" (UniqueName: \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.510533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.510438 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.510681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.510626 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6h4\" (UniqueName: \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.510759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.510742 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574ng\" (UniqueName: \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.510870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.510854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.511245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.511182 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.511586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.511564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.515013 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.514987 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:00.529616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.527153 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:00.540147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.535267 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574ng\" (UniqueName: \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.542144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.542119 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6h4\" (UniqueName: \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.595435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.595402 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:00.599821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.599789 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:00.864834 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.864796 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:00.877864 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:45:00.877831 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac7243_a0b2_49c6_b573_1e2217348a5d.slice/crio-d84c5ee73a94673a144e28b88bac143ed2c8fd5b1ef310872c09d921a3b895db WatchSource:0}: Error finding container d84c5ee73a94673a144e28b88bac143ed2c8fd5b1ef310872c09d921a3b895db: Status 404 returned error can't find the container with id d84c5ee73a94673a144e28b88bac143ed2c8fd5b1ef310872c09d921a3b895db Oct 31 00:45:00.878057 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.878040 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:00.953743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.953716 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:00.985136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:00.985109 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:01.803878 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.803781 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerStarted","Data":"731dbb234182801bafd6024f2a33f3a9379dac192494da9e04db57d14516c31f"} Oct 31 00:45:01.803878 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.803846 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerStarted","Data":"e29919c626f61b0e58eda5ab29e832b999315f2dc44ba8f4d8e422bb167fad34"} Oct 31 00:45:01.805840 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.805814 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerStarted","Data":"b03f1868fa1cde2817e353481bab95d28f43c59f427489b3e0b1eb1a7bd4cf60"} Oct 31 00:45:01.805974 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.805862 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerStarted","Data":"d84c5ee73a94673a144e28b88bac143ed2c8fd5b1ef310872c09d921a3b895db"} Oct 31 00:45:01.807636 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.807612 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerStarted","Data":"32e1261a023b7699bc4de3242b1b2ec6e320de586ba0a8b59c8c7d4329d4d24d"} Oct 31 00:45:01.807725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.807658 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerStarted","Data":"7a988e9e92c325a802ce1e84eb39beac0de208e938965fecbdfd5d56f3314d40"} Oct 31 00:45:01.809525 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.809497 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerStarted","Data":"47410b46072916fc1560483acdd9c35e5404ce3c80642de343503e6af78c6a29"} Oct 31 00:45:01.809630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.809549 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerStarted","Data":"8fcfac351cee660f848bf145a51ad26a6188d7c355b81df7351ea155eb634265"} Oct 31 00:45:01.842374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.842312 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" podStartSLOduration=1.8422946599999999 podStartE2EDuration="1.84229466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.840755721 +0000 UTC m=+1570.289843362" watchObservedRunningTime="2026-10-31 00:45:01.84229466 +0000 UTC m=+1570.291382300" Oct 31 00:45:01.880553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.880506 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" podStartSLOduration=1.8804932920000001 podStartE2EDuration="1.880493292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.877903098 +0000 UTC m=+1570.326990738" watchObservedRunningTime="2026-10-31 00:45:01.880493292 +0000 UTC m=+1570.329580930" Oct 31 00:45:01.922295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.922244 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" podStartSLOduration=1.922227251 podStartE2EDuration="1.922227251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.916176031 +0000 UTC m=+1570.365263673" watchObservedRunningTime="2026-10-31 00:45:01.922227251 +0000 UTC m=+1570.371314886" Oct 31 00:45:01.951665 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:01.951631 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:45:01.956729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:01.956690 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" podStartSLOduration=1.9566772380000002 podStartE2EDuration="1.956677238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.955466485 +0000 UTC m=+1570.404554125" watchObservedRunningTime="2026-10-31 00:45:01.956677238 +0000 UTC m=+1570.405764904" Oct 31 00:45:02.771265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.771233 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:45:02.781728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.781688 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 00:45:02.824095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.824027 5011 generic.go:356] "Generic (PLEG): container finished" podID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerID="731dbb234182801bafd6024f2a33f3a9379dac192494da9e04db57d14516c31f" exitCode=60 Oct 31 00:45:02.824768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.824437 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerDied","Data":"731dbb234182801bafd6024f2a33f3a9379dac192494da9e04db57d14516c31f"} Oct 31 00:45:02.824920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.824899 5011 scope.go:122] "RemoveContainer" containerID="731dbb234182801bafd6024f2a33f3a9379dac192494da9e04db57d14516c31f" Oct 31 00:45:02.827398 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.827375 5011 generic.go:356] "Generic (PLEG): container finished" podID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerID="b03f1868fa1cde2817e353481bab95d28f43c59f427489b3e0b1eb1a7bd4cf60" exitCode=60 Oct 31 00:45:02.827619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.827421 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerDied","Data":"b03f1868fa1cde2817e353481bab95d28f43c59f427489b3e0b1eb1a7bd4cf60"} Oct 31 00:45:02.827789 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.827770 5011 scope.go:122] "RemoveContainer" containerID="b03f1868fa1cde2817e353481bab95d28f43c59f427489b3e0b1eb1a7bd4cf60" Oct 31 00:45:02.839849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.839825 5011 generic.go:356] "Generic (PLEG): container finished" podID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerID="32e1261a023b7699bc4de3242b1b2ec6e320de586ba0a8b59c8c7d4329d4d24d" exitCode=60 Oct 31 00:45:02.840700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.840684 5011 scope.go:122] "RemoveContainer" containerID="32e1261a023b7699bc4de3242b1b2ec6e320de586ba0a8b59c8c7d4329d4d24d" Oct 31 00:45:02.840818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.840804 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerDied","Data":"32e1261a023b7699bc4de3242b1b2ec6e320de586ba0a8b59c8c7d4329d4d24d"} Oct 31 00:45:02.850292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.850258 5011 generic.go:356] "Generic (PLEG): container finished" podID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerID="47410b46072916fc1560483acdd9c35e5404ce3c80642de343503e6af78c6a29" exitCode=60 Oct 31 00:45:02.850421 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.850322 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerDied","Data":"47410b46072916fc1560483acdd9c35e5404ce3c80642de343503e6af78c6a29"} Oct 31 00:45:02.850713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:02.850691 5011 scope.go:122] "RemoveContainer" containerID="47410b46072916fc1560483acdd9c35e5404ce3c80642de343503e6af78c6a29" Oct 31 00:45:03.863621 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.863588 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerStarted","Data":"fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841"} Oct 31 00:45:03.865761 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.865737 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerStarted","Data":"8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1"} Oct 31 00:45:03.867667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.867644 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerStarted","Data":"39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42"} Oct 31 00:45:03.869424 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.869399 5011 generic.go:356] "Generic (PLEG): container finished" podID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerID="4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd" exitCode=60 Oct 31 00:45:03.869547 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.869446 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerDied","Data":"4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd"} Oct 31 00:45:03.869547 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.869494 5011 scope.go:122] "RemoveContainer" containerID="32e1261a023b7699bc4de3242b1b2ec6e320de586ba0a8b59c8c7d4329d4d24d" Oct 31 00:45:03.869740 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:03.869717 5011 scope.go:122] "RemoveContainer" containerID="4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd" Oct 31 00:45:03.870089 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:03.870065 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr_odh-ai-gateway-infra(894678dc-069e-493d-8a3d-2b4f2a2b7d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" Oct 31 00:45:04.882637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.882604 5011 generic.go:356] "Generic (PLEG): container finished" podID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerID="8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1" exitCode=60 Oct 31 00:45:04.883073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.882691 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerDied","Data":"8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1"} Oct 31 00:45:04.883073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.882800 5011 scope.go:122] "RemoveContainer" containerID="731dbb234182801bafd6024f2a33f3a9379dac192494da9e04db57d14516c31f" Oct 31 00:45:04.883073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.882955 5011 scope.go:122] "RemoveContainer" containerID="8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1" Oct 31 00:45:04.883377 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:04.883353 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt_odh-ai-gateway-infra(5193bf51-7fa2-4be3-9cd5-36ad5204bb73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" Oct 31 00:45:04.884910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.884891 5011 generic.go:356] "Generic (PLEG): container finished" podID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerID="39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42" exitCode=60 Oct 31 00:45:04.884998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.884956 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerDied","Data":"39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42"} Oct 31 00:45:04.885201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.885186 5011 scope.go:122] "RemoveContainer" containerID="39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42" Oct 31 00:45:04.885572 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:04.885553 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk_odh-ai-gateway-infra(48ac7243-a0b2-49c6-b573-1e2217348a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" Oct 31 00:45:04.886911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.886891 5011 scope.go:122] "RemoveContainer" containerID="4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd" Oct 31 00:45:04.887194 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:04.887170 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr_odh-ai-gateway-infra(894678dc-069e-493d-8a3d-2b4f2a2b7d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" Oct 31 00:45:04.888819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.888795 5011 generic.go:356] "Generic (PLEG): container finished" podID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerID="fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841" exitCode=60 Oct 31 00:45:04.888907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.888829 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerDied","Data":"fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841"} Oct 31 00:45:04.889091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.889075 5011 scope.go:122] "RemoveContainer" containerID="fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841" Oct 31 00:45:04.889398 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:04.889377 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll_odh-ai-gateway-infra(b5789ff3-eb54-416a-81f8-dcd5dabeeefe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" Oct 31 00:45:04.897255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.897199 5011 scope.go:122] "RemoveContainer" containerID="b03f1868fa1cde2817e353481bab95d28f43c59f427489b3e0b1eb1a7bd4cf60" Oct 31 00:45:04.917942 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:04.917922 5011 scope.go:122] "RemoveContainer" containerID="47410b46072916fc1560483acdd9c35e5404ce3c80642de343503e6af78c6a29" Oct 31 00:45:05.902125 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:05.902095 5011 scope.go:122] "RemoveContainer" containerID="fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841" Oct 31 00:45:05.902779 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:05.902576 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll_odh-ai-gateway-infra(b5789ff3-eb54-416a-81f8-dcd5dabeeefe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" Oct 31 00:45:05.906302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:05.906277 5011 scope.go:122] "RemoveContainer" containerID="8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1" Oct 31 00:45:05.906946 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:05.906913 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt_odh-ai-gateway-infra(5193bf51-7fa2-4be3-9cd5-36ad5204bb73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" Oct 31 00:45:05.909845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:05.909825 5011 scope.go:122] "RemoveContainer" containerID="39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42" Oct 31 00:45:05.910080 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:05.910062 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk_odh-ai-gateway-infra(48ac7243-a0b2-49c6-b573-1e2217348a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" Oct 31 00:45:10.037122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:10.037076 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k"] Oct 31 00:45:10.043890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:10.043860 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2798879a-kg22k"] Oct 31 00:45:11.684815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:11.684771 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefe13aa-dfb5-4141-83c6-9575aff00a74" path="/var/lib/kubelet/pods/fefe13aa-dfb5-4141-83c6-9575aff00a74/volumes" Oct 31 00:45:14.675647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:14.675618 5011 scope.go:122] "RemoveContainer" containerID="fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841" Oct 31 00:45:15.665385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:15.665345 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:45:15.693770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:15.693439 5011 scope.go:122] "RemoveContainer" containerID="8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1" Oct 31 00:45:15.826685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:15.819625 5011 scope.go:122] "RemoveContainer" containerID="4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd" Oct 31 00:45:16.015775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:16.015744 5011 generic.go:356] "Generic (PLEG): container finished" podID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerID="8b890716eb0f71bb258826a9e1ac6cde3ddac74f6c4c59a0efaaa68510e41d4e" exitCode=60 Oct 31 00:45:16.015940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:16.015816 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerDied","Data":"8b890716eb0f71bb258826a9e1ac6cde3ddac74f6c4c59a0efaaa68510e41d4e"} Oct 31 00:45:16.015940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:16.015929 5011 scope.go:122] "RemoveContainer" containerID="fd165f85e9133e3dab273a42da320c0dd4740f71fd04a84118249d4caac2f841" Oct 31 00:45:16.016383 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:16.016363 5011 scope.go:122] "RemoveContainer" containerID="8b890716eb0f71bb258826a9e1ac6cde3ddac74f6c4c59a0efaaa68510e41d4e" Oct 31 00:45:16.016757 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:16.016725 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll_odh-ai-gateway-infra(b5789ff3-eb54-416a-81f8-dcd5dabeeefe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" Oct 31 00:45:17.087293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.083859 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:17.128598 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.124868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerStarted","Data":"af72058ecf3e9edccf3b51bb5747086329eb6a42160183fbe050b0c09c9614f1"} Oct 31 00:45:17.131695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.130181 5011 generic.go:356] "Generic (PLEG): container finished" podID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerID="f3e51a1e0d2ee837203a1d1b0b2f5d5b81d8714044edf447c42092860ebb5c26" exitCode=60 Oct 31 00:45:17.131695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.130383 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerDied","Data":"f3e51a1e0d2ee837203a1d1b0b2f5d5b81d8714044edf447c42092860ebb5c26"} Oct 31 00:45:17.131695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.130438 5011 scope.go:122] "RemoveContainer" containerID="4bee1e193c912fea86b53372b87b226b16a306ab2c3ee1decb724ae6d1d7f9fd" Oct 31 00:45:17.131695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.130890 5011 scope.go:122] "RemoveContainer" containerID="f3e51a1e0d2ee837203a1d1b0b2f5d5b81d8714044edf447c42092860ebb5c26" Oct 31 00:45:17.131695 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:17.131336 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr_odh-ai-gateway-infra(894678dc-069e-493d-8a3d-2b4f2a2b7d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" Oct 31 00:45:17.766009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.761903 5011 scope.go:122] "RemoveContainer" containerID="39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42" Oct 31 00:45:17.976229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:17.976162 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:18.501450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.501409 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.535235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.530315 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.535235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.530608 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:45:18.537341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.535934 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\") pod \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " Oct 31 00:45:18.537341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.536061 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\" (UniqueName: \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\") pod \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\" (UID: \"b5789ff3-eb54-416a-81f8-dcd5dabeeefe\") " Oct 31 00:45:18.547141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.547109 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca" pod "b5789ff3-eb54-416a-81f8-dcd5dabeeefe" (UID: "b5789ff3-eb54-416a-81f8-dcd5dabeeefe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.552477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.551100 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:18.606237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.597937 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:18.831232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.830180 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:18.871612 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.853337 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.913021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.855996 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.914278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.856034 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.977314 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.966372 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:45:18.977314 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:18.967408 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:45:19.008267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.008220 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.009158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.009116 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.017242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.009701 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 00:45:19.017497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.017469 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 31 00:45:19.049628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.048629 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4" pod "b5789ff3-eb54-416a-81f8-dcd5dabeeefe" (UID: "b5789ff3-eb54-416a-81f8-dcd5dabeeefe"). InnerVolumeSpecName "kube-api-access-jp6h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.065416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.062653 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.065416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.065340 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.116293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.116071 5011 generic.go:356] "Generic (PLEG): container finished" podID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerID="af72058ecf3e9edccf3b51bb5747086329eb6a42160183fbe050b0c09c9614f1" exitCode=60 Oct 31 00:45:19.116293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.116251 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerDied","Data":"af72058ecf3e9edccf3b51bb5747086329eb6a42160183fbe050b0c09c9614f1"} Oct 31 00:45:19.116529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.116319 5011 scope.go:122] "RemoveContainer" containerID="8f64e6ddcb44991f99f406f0fd6417332326e40498adfaf6a63b9fe5201bd9e1" Oct 31 00:45:19.172017 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.171980 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6h4\" (UniqueName: \"kubernetes.io/projected/b5789ff3-eb54-416a-81f8-dcd5dabeeefe-kube-api-access-jp6h4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:19.227814 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.227665 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" event={"ID":"b5789ff3-eb54-416a-81f8-dcd5dabeeefe","Type":"ContainerDied","Data":"8fcfac351cee660f848bf145a51ad26a6188d7c355b81df7351ea155eb634265"} Oct 31 00:45:19.228785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.228759 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll" Oct 31 00:45:19.408800 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.406420 5011 scope.go:122] "RemoveContainer" containerID="8b890716eb0f71bb258826a9e1ac6cde3ddac74f6c4c59a0efaaa68510e41d4e" Oct 31 00:45:19.490229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.485427 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:45:19.566569 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.566529 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.571193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.570930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:45:19.589291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.588388 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:45:19.589291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.588461 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:45:19.622552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.622297 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.856234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.797250 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:45:19.918368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.918317 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:45:19.944231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:19.934365 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.093413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.070338 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:45:20.114979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.104279 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:45:20.114979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.106055 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:45:20.116098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.115334 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:45:20.116098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.115671 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220592 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220680 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220727 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220760 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220792 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220827 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220863 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220895 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:45:20.223697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.220925 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 00:45:20.284245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.280256 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.382299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.382262 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890125-vcwll"] Oct 31 00:45:20.560236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.554910 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:45:20.561084 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.561055 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:45:20.575360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.575323 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 00:45:20.596351 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.596313 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:45:20.857233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.856084 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.882787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.882749 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:20.925391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.925313 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.959403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:20.959357 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.033452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.033409 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerStarted","Data":"96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0"} Oct 31 00:45:21.039238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.034152 5011 scope.go:122] "RemoveContainer" containerID="96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0" Oct 31 00:45:21.040062 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:21.040017 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk_odh-ai-gateway-infra(48ac7243-a0b2-49c6-b573-1e2217348a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" Oct 31 00:45:21.070393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.070359 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\" (UniqueName: \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\") pod \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " Oct 31 00:45:21.070694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.070665 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\") pod \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\" (UID: \"894678dc-069e-493d-8a3d-2b4f2a2b7d77\") " Oct 31 00:45:21.093757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.093720 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca" pod "894678dc-069e-493d-8a3d-2b4f2a2b7d77" (UID: "894678dc-069e-493d-8a3d-2b4f2a2b7d77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:21.117482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.117381 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:45:21.125145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.125087 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.160306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.160269 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 00:45:21.161926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.161895 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz" pod "894678dc-069e-493d-8a3d-2b4f2a2b7d77" (UID: "894678dc-069e-493d-8a3d-2b4f2a2b7d77"). InnerVolumeSpecName "kube-api-access-kfbmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:21.163305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.163278 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:45:21.175441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.175375 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.217436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.217400 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbmz\" (UniqueName: \"kubernetes.io/projected/894678dc-069e-493d-8a3d-2b4f2a2b7d77-kube-api-access-kfbmz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:21.217681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.217660 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894678dc-069e-493d-8a3d-2b4f2a2b7d77-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:21.293232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.268261 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.582239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.577517 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:45:21.582239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.579036 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:45:21.582239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.580184 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.586694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.586657 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:21.586836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.586710 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.632914 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.631779 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:45:21.648680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.647076 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 00:45:21.704990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.692540 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:21.704990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.692603 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.735841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.735800 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:45:21.834557 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:21.834440 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac7243_a0b2_49c6_b573_1e2217348a5d.slice/crio-conmon-96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac7243_a0b2_49c6_b573_1e2217348a5d.slice/crio-96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:45:21.956718 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:21.956588 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 00:45:22.053233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.032316 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:45:22.310174 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.310125 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:45:22.410283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.406414 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:45:22.750727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.750622 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:22.812098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.812056 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 00:45:22.821237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.813595 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" path="/var/lib/kubelet/pods/b5789ff3-eb54-416a-81f8-dcd5dabeeefe/volumes" Oct 31 00:45:22.980265 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:22.974840 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.282s" Oct 31 00:45:22.980265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.975166 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:45:23.069313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.066596 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.086372 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.077690 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.100666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.843341 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:45:23.117293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:22.951870 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.101537 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.102382 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.166475 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.131679 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.165914 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 00:45:23.178306 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.166153 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.297298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.297253 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:45:23.370044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.354379 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.370044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.355153 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:45:23.699581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.698021 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 00:45:23.760997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.760947 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.766441 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.766415 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:23.800817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.800763 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" event={"ID":"5193bf51-7fa2-4be3-9cd5-36ad5204bb73","Type":"ContainerDied","Data":"e29919c626f61b0e58eda5ab29e832b999315f2dc44ba8f4d8e422bb167fad34"} Oct 31 00:45:23.801103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.801069 5011 scope.go:122] "RemoveContainer" containerID="af72058ecf3e9edccf3b51bb5747086329eb6a42160183fbe050b0c09c9614f1" Oct 31 00:45:23.853634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.820832 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:23.907968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.851164 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 31 00:45:23.908390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.901670 5011 generic.go:356] "Generic (PLEG): container finished" podID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerID="96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0" exitCode=60 Oct 31 00:45:23.908868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.901697 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerDied","Data":"96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0"} Oct 31 00:45:23.909568 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.909553 5011 scope.go:122] "RemoveContainer" containerID="96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0" Oct 31 00:45:23.910164 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:23.910141 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk_odh-ai-gateway-infra(48ac7243-a0b2-49c6-b573-1e2217348a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" Oct 31 00:45:23.910637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.903962 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\") pod \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " Oct 31 00:45:23.910997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.910979 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\" (UniqueName: \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\") pod \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\" (UID: \"5193bf51-7fa2-4be3-9cd5-36ad5204bb73\") " Oct 31 00:45:23.912474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.904359 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca" pod "5193bf51-7fa2-4be3-9cd5-36ad5204bb73" (UID: "5193bf51-7fa2-4be3-9cd5-36ad5204bb73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:23.927259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.919741 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" event={"ID":"894678dc-069e-493d-8a3d-2b4f2a2b7d77","Type":"ContainerDied","Data":"7a988e9e92c325a802ce1e84eb39beac0de208e938965fecbdfd5d56f3314d40"} Oct 31 00:45:23.975278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.951850 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr" Oct 31 00:45:23.995258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.993832 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng" pod "5193bf51-7fa2-4be3-9cd5-36ad5204bb73" (UID: "5193bf51-7fa2-4be3-9cd5-36ad5204bb73"). InnerVolumeSpecName "kube-api-access-574ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:23.995258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:23.994725 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 00:45:24.029458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.029430 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 00:45:24.079171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.062650 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:45:24.079171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.062714 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:45:24.079171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.065248 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574ng\" (UniqueName: \"kubernetes.io/projected/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-kube-api-access-574ng\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:24.079171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.065275 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5193bf51-7fa2-4be3-9cd5-36ad5204bb73-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:24.093200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.093172 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 00:45:24.109298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.099021 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:45:24.109298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.101146 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 00:45:24.127231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.110290 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 00:45:24.189231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.127567 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:24.189684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.189655 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:45:24.189858 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.189847 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 00:45:24.190038 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.114622 5011 scope.go:122] "RemoveContainer" containerID="39d8929bc2a61b5ffede191e816693e4382b2043e1869bec6f13742e6d47ef42" Oct 31 00:45:24.289837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.289805 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:45:24.409059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.409025 5011 scope.go:122] "RemoveContainer" containerID="f3e51a1e0d2ee837203a1d1b0b2f5d5b81d8714044edf447c42092860ebb5c26" Oct 31 00:45:24.685236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.684381 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:24.699168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.699138 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890125-hllvr"] Oct 31 00:45:24.971283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:24.970380 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt" Oct 31 00:45:25.060844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.059713 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:25.067510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.067455 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890125-gtkvt"] Oct 31 00:45:25.207993 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.207969 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:25.371711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.371674 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\") pod \"48ac7243-a0b2-49c6-b573-1e2217348a5d\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " Oct 31 00:45:25.371975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.371954 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\" (UniqueName: \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\") pod \"48ac7243-a0b2-49c6-b573-1e2217348a5d\" (UID: \"48ac7243-a0b2-49c6-b573-1e2217348a5d\") " Oct 31 00:45:25.372097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.372073 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca" pod "48ac7243-a0b2-49c6-b573-1e2217348a5d" (UID: "48ac7243-a0b2-49c6-b573-1e2217348a5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:25.372699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.372673 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac7243-a0b2-49c6-b573-1e2217348a5d-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:25.374287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.374255 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72" pod "48ac7243-a0b2-49c6-b573-1e2217348a5d" (UID: "48ac7243-a0b2-49c6-b573-1e2217348a5d"). InnerVolumeSpecName "kube-api-access-7jp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:25.474581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.474540 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jp72\" (UniqueName: \"kubernetes.io/projected/48ac7243-a0b2-49c6-b573-1e2217348a5d-kube-api-access-7jp72\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:25.684966 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.684873 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" path="/var/lib/kubelet/pods/5193bf51-7fa2-4be3-9cd5-36ad5204bb73/volumes" Oct 31 00:45:25.685480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.685457 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" path="/var/lib/kubelet/pods/894678dc-069e-493d-8a3d-2b4f2a2b7d77/volumes" Oct 31 00:45:25.988690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.988603 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" Oct 31 00:45:25.989137 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.988693 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk" event={"ID":"48ac7243-a0b2-49c6-b573-1e2217348a5d","Type":"ContainerDied","Data":"d84c5ee73a94673a144e28b88bac143ed2c8fd5b1ef310872c09d921a3b895db"} Oct 31 00:45:25.989137 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:25.988743 5011 scope.go:122] "RemoveContainer" containerID="96d0fd69292a62ef6757d51784a2b5c7b017038efd608fe07cb5923ed47d8de0" Oct 31 00:45:26.056171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:26.056131 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s"] Oct 31 00:45:26.062682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:26.062647 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89025758-rxc7s"] Oct 31 00:45:26.082946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:26.082909 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:26.089073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:26.089044 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890125-sntzk"] Oct 31 00:45:27.035305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.035171 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg"] Oct 31 00:45:27.044313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.044273 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42319158-zhqpg"] Oct 31 00:45:27.687622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.687591 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116b894a-8e1b-4ca6-a026-43eb9b62ea77" path="/var/lib/kubelet/pods/116b894a-8e1b-4ca6-a026-43eb9b62ea77/volumes" Oct 31 00:45:27.688102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.688088 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" path="/var/lib/kubelet/pods/48ac7243-a0b2-49c6-b573-1e2217348a5d/volumes" Oct 31 00:45:27.688593 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.688579 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62b99a5-19e7-41f4-912d-f888f9773aee" path="/var/lib/kubelet/pods/d62b99a5-19e7-41f4-912d-f888f9773aee/volumes" Oct 31 00:45:27.876402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.876350 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:27.876580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:27.876416 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:29.334815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.334745 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" podUID="6792620a-d102-4a41-be55-52e46d12adf1" containerName="payload-pre-processing" containerID="cri-o://1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68" gracePeriod=25 Oct 31 00:45:29.675254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.675223 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:45:29.768575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.768520 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\" (UniqueName: \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\") pod \"6792620a-d102-4a41-be55-52e46d12adf1\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " Oct 31 00:45:29.768891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.768868 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\") pod \"6792620a-d102-4a41-be55-52e46d12adf1\" (UID: \"6792620a-d102-4a41-be55-52e46d12adf1\") " Oct 31 00:45:29.769591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.769486 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume" pod "6792620a-d102-4a41-be55-52e46d12adf1" (UID: "6792620a-d102-4a41-be55-52e46d12adf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:29.771725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.771701 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6792620a-d102-4a41-be55-52e46d12adf1-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:29.776404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.776377 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5" pod "6792620a-d102-4a41-be55-52e46d12adf1" (UID: "6792620a-d102-4a41-be55-52e46d12adf1"). InnerVolumeSpecName "kube-api-access-fb4w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:29.875436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:29.875393 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb4w5\" (UniqueName: \"kubernetes.io/projected/6792620a-d102-4a41-be55-52e46d12adf1-kube-api-access-fb4w5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:30.040917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.040795 5011 generic.go:356] "Generic (PLEG): container finished" podID="6792620a-d102-4a41-be55-52e46d12adf1" containerID="1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68" exitCode=0 Oct 31 00:45:30.040917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.040840 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" Oct 31 00:45:30.041318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.040966 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" event={"ID":"6792620a-d102-4a41-be55-52e46d12adf1","Type":"ContainerDied","Data":"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68"} Oct 31 00:45:30.041423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.041370 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh" event={"ID":"6792620a-d102-4a41-be55-52e46d12adf1","Type":"ContainerDied","Data":"d05d77845540693f43b24cb69d5e9d051e14a34c106248f80423b0de984c63c1"} Oct 31 00:45:30.041423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.041414 5011 scope.go:122] "RemoveContainer" containerID="1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68" Oct 31 00:45:30.057089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.057070 5011 scope.go:122] "RemoveContainer" containerID="1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68" Oct 31 00:45:30.057541 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:30.057511 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68\": container with ID starting with 1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68 not found: ID does not exist" containerID="1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68" Oct 31 00:45:30.057653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.057551 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68"} err="failed to get container status \"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68\": rpc error: code = NotFound desc = could not find container \"1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68\": container with ID starting with 1ab0a22ca9bfdc8aa2e4ff553a501b20f7a24161403c73efcf6f4c2e38aa9d68 not found: ID does not exist" Oct 31 00:45:30.093347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.093320 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:45:30.107388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:30.107350 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d66e61c1-7bd4c4589f-8gzxh"] Oct 31 00:45:31.034154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.034081 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w"] Oct 31 00:45:31.045019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.044986 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m"] Oct 31 00:45:31.050476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.050447 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8258fa1-kdl8w"] Oct 31 00:45:31.055539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.055511 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ab8e7c-m8h6m"] Oct 31 00:45:31.057757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.057729 5011 generic.go:356] "Generic (PLEG): container finished" podID="267960c0-24ec-422e-bde3-4068242dff0c" containerID="3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351" exitCode=0 Oct 31 00:45:31.057886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.057809 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerDied","Data":"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351"} Oct 31 00:45:31.058371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.058352 5011 scope.go:122] "RemoveContainer" containerID="3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351" Oct 31 00:45:31.060078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.060050 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:31.060175 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.060113 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:31.060175 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.060144 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:31.684831 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.684795 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4873fe09-d4ba-43e9-b1a3-5db525b3ec69" path="/var/lib/kubelet/pods/4873fe09-d4ba-43e9-b1a3-5db525b3ec69/volumes" Oct 31 00:45:31.685347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.685332 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6792620a-d102-4a41-be55-52e46d12adf1" path="/var/lib/kubelet/pods/6792620a-d102-4a41-be55-52e46d12adf1/volumes" Oct 31 00:45:31.685769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:31.685756 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54fab01-0d15-4316-b6b8-370069bd01e9" path="/var/lib/kubelet/pods/f54fab01-0d15-4316-b6b8-370069bd01e9/volumes" Oct 31 00:45:31.978282 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:31.978179 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:45:32.068810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:32.068765 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerStarted","Data":"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1"} Oct 31 00:45:32.575152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:32.575118 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:32.575390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:32.575363 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:32.575440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:32.575418 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:33.575175 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:33.575144 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:33.575595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:33.575225 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:33.877528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:33.877457 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:45:33.877804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:33.877775 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" podUID="243abd72-5ae5-4b9d-a741-ec02d42b89db" containerName="maas-api" containerID="cri-o://26f518f42d939dcc8b9bb8230449ef50afc764745adb12225fc18440f4103b16" gracePeriod=30 Oct 31 00:45:34.076035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.075966 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f"] Oct 31 00:45:34.131289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.128625 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34cb65cd-d5w7f"] Oct 31 00:45:34.262378 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.261400 5011 generic.go:356] "Generic (PLEG): container finished" podID="243abd72-5ae5-4b9d-a741-ec02d42b89db" containerID="26f518f42d939dcc8b9bb8230449ef50afc764745adb12225fc18440f4103b16" exitCode=0 Oct 31 00:45:34.262378 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.261652 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" event={"ID":"243abd72-5ae5-4b9d-a741-ec02d42b89db","Type":"ContainerDied","Data":"26f518f42d939dcc8b9bb8230449ef50afc764745adb12225fc18440f4103b16"} Oct 31 00:45:34.575188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.575151 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:34.575844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.575810 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:34.905379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:34.905347 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:45:35.047540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.047501 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\") pod \"243abd72-5ae5-4b9d-a741-ec02d42b89db\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " Oct 31 00:45:35.047540 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.047547 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\") pod \"243abd72-5ae5-4b9d-a741-ec02d42b89db\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " Oct 31 00:45:35.047812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.047628 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") pod \"243abd72-5ae5-4b9d-a741-ec02d42b89db\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " Oct 31 00:45:35.047812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.047686 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\" (UniqueName: \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\") pod \"243abd72-5ae5-4b9d-a741-ec02d42b89db\" (UID: \"243abd72-5ae5-4b9d-a741-ec02d42b89db\") " Oct 31 00:45:35.048043 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.047947 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca" pod "243abd72-5ae5-4b9d-a741-ec02d42b89db" (UID: "243abd72-5ae5-4b9d-a741-ec02d42b89db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:35.048414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.048294 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/243abd72-5ae5-4b9d-a741-ec02d42b89db-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:35.050283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.050244 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx" pod "243abd72-5ae5-4b9d-a741-ec02d42b89db" (UID: "243abd72-5ae5-4b9d-a741-ec02d42b89db"). InnerVolumeSpecName "kube-api-access-zpjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:35.050561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.050528 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs" pod "243abd72-5ae5-4b9d-a741-ec02d42b89db" (UID: "243abd72-5ae5-4b9d-a741-ec02d42b89db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:35.050798 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.050774 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls" pod "243abd72-5ae5-4b9d-a741-ec02d42b89db" (UID: "243abd72-5ae5-4b9d-a741-ec02d42b89db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:35.150452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.150143 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:35.150452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.150176 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/243abd72-5ae5-4b9d-a741-ec02d42b89db-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:35.150452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.150186 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjpx\" (UniqueName: \"kubernetes.io/projected/243abd72-5ae5-4b9d-a741-ec02d42b89db-kube-api-access-zpjpx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:35.274812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.274773 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" event={"ID":"243abd72-5ae5-4b9d-a741-ec02d42b89db","Type":"ContainerDied","Data":"1639cbeedf3f2f3878cba18a272b18e8cf818a67cf2ce3f72ccf082864870923"} Oct 31 00:45:35.274812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.274801 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9" Oct 31 00:45:35.275091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.274847 5011 scope.go:122] "RemoveContainer" containerID="26f518f42d939dcc8b9bb8230449ef50afc764745adb12225fc18440f4103b16" Oct 31 00:45:35.348492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.348452 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:45:35.354386 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.354359 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d66e61c1-755b844887-tjsp9"] Oct 31 00:45:35.574989 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.574946 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:35.575189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.575007 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:35.634230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.633413 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:45:35.696033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.694825 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243abd72-5ae5-4b9d-a741-ec02d42b89db" path="/var/lib/kubelet/pods/243abd72-5ae5-4b9d-a741-ec02d42b89db/volumes" Oct 31 00:45:35.696033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:35.695632 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85793bc6-c88d-461c-a170-9fe82a64e010" path="/var/lib/kubelet/pods/85793bc6-c88d-461c-a170-9fe82a64e010/volumes" Oct 31 00:45:36.579269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:36.578264 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:36.579269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:36.578329 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:37.777249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:37.775592 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:37.777249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:37.776715 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:37.778603 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:37.778567 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:38.088305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:38.088270 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c"] Oct 31 00:45:38.117083 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:38.117016 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7494082-sxh6c"] Oct 31 00:45:39.686054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:39.686008 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d" path="/var/lib/kubelet/pods/af03f0ce-2c87-48f3-a8d7-4c9eb8d1c29d/volumes" Oct 31 00:45:40.047482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:40.047386 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr"] Oct 31 00:45:40.054515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:40.054481 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e2a215-5ckbr"] Oct 31 00:45:40.634569 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:40.634486 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" podUID="f643fb1a-8434-4331-94ed-f39dceddd005" containerName="payload-pre-processing" containerID="cri-o://f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab" gracePeriod=25 Oct 31 00:45:40.927910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:40.927881 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:45:41.015694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.015655 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\" (UniqueName: \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\") pod \"f643fb1a-8434-4331-94ed-f39dceddd005\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " Oct 31 00:45:41.015895 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.015809 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\") pod \"f643fb1a-8434-4331-94ed-f39dceddd005\" (UID: \"f643fb1a-8434-4331-94ed-f39dceddd005\") " Oct 31 00:45:41.016282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.016223 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume" pod "f643fb1a-8434-4331-94ed-f39dceddd005" (UID: "f643fb1a-8434-4331-94ed-f39dceddd005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:41.016609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.016590 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f643fb1a-8434-4331-94ed-f39dceddd005-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:41.017735 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.017710 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj" pod "f643fb1a-8434-4331-94ed-f39dceddd005" (UID: "f643fb1a-8434-4331-94ed-f39dceddd005"). InnerVolumeSpecName "kube-api-access-8dzfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:41.117806 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.117776 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzfj\" (UniqueName: \"kubernetes.io/projected/f643fb1a-8434-4331-94ed-f39dceddd005-kube-api-access-8dzfj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:41.345379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.345347 5011 generic.go:356] "Generic (PLEG): container finished" podID="f643fb1a-8434-4331-94ed-f39dceddd005" containerID="f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab" exitCode=0 Oct 31 00:45:41.345563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.345391 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" Oct 31 00:45:41.345563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.345437 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" event={"ID":"f643fb1a-8434-4331-94ed-f39dceddd005","Type":"ContainerDied","Data":"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab"} Oct 31 00:45:41.345563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.345498 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww" event={"ID":"f643fb1a-8434-4331-94ed-f39dceddd005","Type":"ContainerDied","Data":"5a4b8d43f0ae5c13c1a7ee37b4e94558d215520bf4c015febeb5d81a282e0ff5"} Oct 31 00:45:41.345563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.345525 5011 scope.go:122] "RemoveContainer" containerID="f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab" Oct 31 00:45:41.356592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.356576 5011 scope.go:122] "RemoveContainer" containerID="f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab" Oct 31 00:45:41.356869 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:41.356840 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab\": container with ID starting with f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab not found: ID does not exist" containerID="f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab" Oct 31 00:45:41.356952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.356878 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab"} err="failed to get container status \"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab\": rpc error: code = NotFound desc = could not find container \"f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab\": container with ID starting with f847bf8bad89f0d3009fce779c4ff224ff1e84fe49137b8ea6e87b6f868272ab not found: ID does not exist" Oct 31 00:45:41.401516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.401489 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:45:41.412447 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.412422 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9461dc0-66cdd56c9-ndkww"] Oct 31 00:45:41.686129 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.686044 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9442c788-be92-4361-a313-763271f76555" path="/var/lib/kubelet/pods/9442c788-be92-4361-a313-763271f76555/volumes" Oct 31 00:45:41.686578 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:41.686561 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f643fb1a-8434-4331-94ed-f39dceddd005" path="/var/lib/kubelet/pods/f643fb1a-8434-4331-94ed-f39dceddd005/volumes" Oct 31 00:45:46.198237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.198186 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:45:46.229714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.229684 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:45:46.265158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.265123 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:45:46.265401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.265361 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" podUID="56306f5e-9d2b-45be-bfb6-58804eb50c08" containerName="maas-api" containerID="cri-o://24d3787cc847edb01420bc6ea7eaf340b0f7d8728d9aceb476685cfdbc91adab" gracePeriod=30 Oct 31 00:45:46.399180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.399126 5011 generic.go:356] "Generic (PLEG): container finished" podID="56306f5e-9d2b-45be-bfb6-58804eb50c08" containerID="24d3787cc847edb01420bc6ea7eaf340b0f7d8728d9aceb476685cfdbc91adab" exitCode=0 Oct 31 00:45:46.399421 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.399400 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" event={"ID":"56306f5e-9d2b-45be-bfb6-58804eb50c08","Type":"ContainerDied","Data":"24d3787cc847edb01420bc6ea7eaf340b0f7d8728d9aceb476685cfdbc91adab"} Oct 31 00:45:46.744465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.744388 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:45:46.850230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.841815 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\") pod \"56306f5e-9d2b-45be-bfb6-58804eb50c08\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " Oct 31 00:45:46.850620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.850582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca" pod "56306f5e-9d2b-45be-bfb6-58804eb50c08" (UID: "56306f5e-9d2b-45be-bfb6-58804eb50c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:46.850871 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.841897 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\") pod \"56306f5e-9d2b-45be-bfb6-58804eb50c08\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " Oct 31 00:45:46.851167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.851147 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") pod \"56306f5e-9d2b-45be-bfb6-58804eb50c08\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " Oct 31 00:45:46.851817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.851790 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\" (UniqueName: \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\") pod \"56306f5e-9d2b-45be-bfb6-58804eb50c08\" (UID: \"56306f5e-9d2b-45be-bfb6-58804eb50c08\") " Oct 31 00:45:46.853470 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.853445 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56306f5e-9d2b-45be-bfb6-58804eb50c08-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:46.873229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.869279 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs" pod "56306f5e-9d2b-45be-bfb6-58804eb50c08" (UID: "56306f5e-9d2b-45be-bfb6-58804eb50c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:46.919035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.918979 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r" pod "56306f5e-9d2b-45be-bfb6-58804eb50c08" (UID: "56306f5e-9d2b-45be-bfb6-58804eb50c08"). InnerVolumeSpecName "kube-api-access-h267r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:46.921148 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.921105 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls" pod "56306f5e-9d2b-45be-bfb6-58804eb50c08" (UID: "56306f5e-9d2b-45be-bfb6-58804eb50c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:46.960396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.955634 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:46.960396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.955671 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56306f5e-9d2b-45be-bfb6-58804eb50c08-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:46.960396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:46.955688 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h267r\" (UniqueName: \"kubernetes.io/projected/56306f5e-9d2b-45be-bfb6-58804eb50c08-kube-api-access-h267r\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:47.410291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.410252 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" Oct 31 00:45:47.410770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.410737 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf" event={"ID":"56306f5e-9d2b-45be-bfb6-58804eb50c08","Type":"ContainerDied","Data":"e5dcc998f10edff995cfabfe7befc6c920e43f03172505918c13da971417faca"} Oct 31 00:45:47.410903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.410812 5011 scope.go:122] "RemoveContainer" containerID="24d3787cc847edb01420bc6ea7eaf340b0f7d8728d9aceb476685cfdbc91adab" Oct 31 00:45:47.461992 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.461954 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:45:47.472447 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.472415 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a9461dc0-57c479fb4c-2lzwf"] Oct 31 00:45:47.591293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.590681 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:45:47.604228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.591973 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" containerID="cri-o://4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1" gracePeriod=30 Oct 31 00:45:47.699114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.699039 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56306f5e-9d2b-45be-bfb6-58804eb50c08" path="/var/lib/kubelet/pods/56306f5e-9d2b-45be-bfb6-58804eb50c08/volumes" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.746850 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq"] Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747869 5011 cpu_manager.go:414] "containerMap: removing container" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747890 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747910 5011 cpu_manager.go:414] "containerMap: removing container" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747919 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747939 5011 cpu_manager.go:414] "containerMap: removing container" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747947 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747961 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6792620a-d102-4a41-be55-52e46d12adf1" containerName="payload-pre-processing" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747969 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6792620a-d102-4a41-be55-52e46d12adf1" containerName="payload-pre-processing" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747984 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f643fb1a-8434-4331-94ed-f39dceddd005" containerName="payload-pre-processing" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.747992 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f643fb1a-8434-4331-94ed-f39dceddd005" containerName="payload-pre-processing" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748019 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748026 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748048 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748055 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748072 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748079 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748090 5011 cpu_manager.go:414] "containerMap: removing container" podUID="243abd72-5ae5-4b9d-a741-ec02d42b89db" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748097 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243abd72-5ae5-4b9d-a741-ec02d42b89db" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748108 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748115 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748133 5011 cpu_manager.go:414] "containerMap: removing container" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748140 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748150 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748158 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748174 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748181 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748194 5011 cpu_manager.go:414] "containerMap: removing container" podUID="56306f5e-9d2b-45be-bfb6-58804eb50c08" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748201 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56306f5e-9d2b-45be-bfb6-58804eb50c08" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748450 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748463 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748476 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748533 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748545 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748556 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="243abd72-5ae5-4b9d-a741-ec02d42b89db" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748568 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748576 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748586 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5789ff3-eb54-416a-81f8-dcd5dabeeefe" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748598 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="56306f5e-9d2b-45be-bfb6-58804eb50c08" containerName="maas-api" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748608 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748619 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748628 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6792620a-d102-4a41-be55-52e46d12adf1" containerName="payload-pre-processing" Oct 31 00:45:47.750232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.748638 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f643fb1a-8434-4331-94ed-f39dceddd005" containerName="payload-pre-processing" Oct 31 00:45:47.755729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.755691 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.769543 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.769514 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:47.769685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.769669 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:45:47.771637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.771614 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq"] Oct 31 00:45:47.809485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.809454 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.809642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.809600 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.809824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.809804 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqsx\" (UniqueName: \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.925337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.925108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqsx\" (UniqueName: \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.925337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.925240 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.925577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.925346 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.926185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.926155 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.934650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.934617 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:47.951779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:47.951741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqsx\" (UniqueName: \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\") pod \"maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:48.073678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:48.073643 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:48.393553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:48.393527 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq"] Oct 31 00:45:48.393962 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:45:48.393938 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb503a3b_5f90_4ba6_ad9d_9487b12a43df.slice/crio-88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b WatchSource:0}: Error finding container 88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b: Status 404 returned error can't find the container with id 88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b Oct 31 00:45:48.423589 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:48.423558 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" event={"ID":"eb503a3b-5f90-4ba6-ad9d-9487b12a43df","Type":"ContainerStarted","Data":"88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b"} Oct 31 00:45:49.043944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:49.043909 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4"] Oct 31 00:45:49.050430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:49.050402 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a70230-k79w4"] Oct 31 00:45:49.435469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:49.435427 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" event={"ID":"eb503a3b-5f90-4ba6-ad9d-9487b12a43df","Type":"ContainerStarted","Data":"b86adb55b67fc28720cffffb3594836af9e03fdb6a266a31cd0adda07f02266c"} Oct 31 00:45:49.475781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:49.475736 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" podStartSLOduration=2.475721918 podStartE2EDuration="2.475721918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:49.473638547 +0000 UTC m=+1617.922726191" watchObservedRunningTime="2026-10-31 00:45:49.475721918 +0000 UTC m=+1617.924809556" Oct 31 00:45:49.685259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:49.685224 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d901f1-e68f-4cc5-9870-d93272905f88" path="/var/lib/kubelet/pods/20d901f1-e68f-4cc5-9870-d93272905f88/volumes" Oct 31 00:45:50.448610 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:50.448570 5011 generic.go:356] "Generic (PLEG): container finished" podID="eb503a3b-5f90-4ba6-ad9d-9487b12a43df" containerID="b86adb55b67fc28720cffffb3594836af9e03fdb6a266a31cd0adda07f02266c" exitCode=0 Oct 31 00:45:50.449103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:50.448661 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" event={"ID":"eb503a3b-5f90-4ba6-ad9d-9487b12a43df","Type":"ContainerDied","Data":"b86adb55b67fc28720cffffb3594836af9e03fdb6a266a31cd0adda07f02266c"} Oct 31 00:45:51.199547 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.199509 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" podUID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" containerName="payload-processing" containerID="cri-o://9b39dac53c003be9ed4a2950c9e3b4670e687e68a2e6c6938bdd45d2c93321c7" gracePeriod=25 Oct 31 00:45:51.229986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.229914 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" podUID="987e4abc-a0c9-47f2-b3a5-b291b358cede" containerName="payload-pre-processing" containerID="cri-o://237eda4d4fba7a5cf8abfcffea03aa8ddd3e927dd1a0beb64021a7fc1e276dbc" gracePeriod=25 Oct 31 00:45:51.633230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.623592 5011 generic.go:356] "Generic (PLEG): container finished" podID="987e4abc-a0c9-47f2-b3a5-b291b358cede" containerID="237eda4d4fba7a5cf8abfcffea03aa8ddd3e927dd1a0beb64021a7fc1e276dbc" exitCode=0 Oct 31 00:45:51.633230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.625888 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" event={"ID":"987e4abc-a0c9-47f2-b3a5-b291b358cede","Type":"ContainerDied","Data":"237eda4d4fba7a5cf8abfcffea03aa8ddd3e927dd1a0beb64021a7fc1e276dbc"} Oct 31 00:45:51.662264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.662230 5011 generic.go:356] "Generic (PLEG): container finished" podID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" containerID="9b39dac53c003be9ed4a2950c9e3b4670e687e68a2e6c6938bdd45d2c93321c7" exitCode=0 Oct 31 00:45:51.662946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.662910 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" event={"ID":"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0","Type":"ContainerDied","Data":"9b39dac53c003be9ed4a2950c9e3b4670e687e68a2e6c6938bdd45d2c93321c7"} Oct 31 00:45:51.817980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.816410 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:45:51.867622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.867595 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:45:51.896064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.896027 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\" (UniqueName: \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\") pod \"987e4abc-a0c9-47f2-b3a5-b291b358cede\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " Oct 31 00:45:51.896247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.896149 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\") pod \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " Oct 31 00:45:51.896247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.896189 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\") pod \"987e4abc-a0c9-47f2-b3a5-b291b358cede\" (UID: \"987e4abc-a0c9-47f2-b3a5-b291b358cede\") " Oct 31 00:45:51.896339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.896267 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\" (UniqueName: \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\") pod \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\" (UID: \"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0\") " Oct 31 00:45:51.898623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.898584 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume" pod "6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" (UID: "6a6c69c0-0b49-46e8-9573-5eac80d0f7a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:51.898794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.898657 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume" pod "987e4abc-a0c9-47f2-b3a5-b291b358cede" (UID: "987e4abc-a0c9-47f2-b3a5-b291b358cede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:51.906025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.905340 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw" pod "987e4abc-a0c9-47f2-b3a5-b291b358cede" (UID: "987e4abc-a0c9-47f2-b3a5-b291b358cede"). InnerVolumeSpecName "kube-api-access-dp6jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:51.906025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.905966 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn" pod "6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" (UID: "6a6c69c0-0b49-46e8-9573-5eac80d0f7a0"). InnerVolumeSpecName "kube-api-access-x65jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:51.957886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.956696 5011 scope.go:122] "RemoveContainer" containerID="cd6d677f6523f54a82c67229b4f7437c7cd1315a6db88ba44d30ed819d36b3d5" Oct 31 00:45:51.999331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.999297 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp6jw\" (UniqueName: \"kubernetes.io/projected/987e4abc-a0c9-47f2-b3a5-b291b358cede-kube-api-access-dp6jw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.999331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.999330 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.999482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.999341 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e4abc-a0c9-47f2-b3a5-b291b358cede-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.999482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:51.999351 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65jn\" (UniqueName: \"kubernetes.io/projected/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0-kube-api-access-x65jn\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.003569 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.003549 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:52.106370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.106290 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\" (UniqueName: \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\") pod \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " Oct 31 00:45:52.106510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.106409 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\") pod \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " Oct 31 00:45:52.106555 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.106523 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\") pod \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\" (UID: \"eb503a3b-5f90-4ba6-ad9d-9487b12a43df\") " Oct 31 00:45:52.107404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.107383 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca" pod "eb503a3b-5f90-4ba6-ad9d-9487b12a43df" (UID: "eb503a3b-5f90-4ba6-ad9d-9487b12a43df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:52.109561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.109535 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx" pod "eb503a3b-5f90-4ba6-ad9d-9487b12a43df" (UID: "eb503a3b-5f90-4ba6-ad9d-9487b12a43df"). InnerVolumeSpecName "kube-api-access-8zqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.109986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.109959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca" pod "eb503a3b-5f90-4ba6-ad9d-9487b12a43df" (UID: "eb503a3b-5f90-4ba6-ad9d-9487b12a43df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:52.241818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.217626 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqsx\" (UniqueName: \"kubernetes.io/projected/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-kube-api-access-8zqsx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.241818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.238347 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.241818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.238391 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb503a3b-5f90-4ba6-ad9d-9487b12a43df-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.686879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.686197 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" event={"ID":"987e4abc-a0c9-47f2-b3a5-b291b358cede","Type":"ContainerDied","Data":"b8f7f6af8d74c8a9cf592e5f89d2480ac198b87ed5829fe7fa96e1f28597bbd9"} Oct 31 00:45:52.688187 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.688164 5011 scope.go:122] "RemoveContainer" containerID="237eda4d4fba7a5cf8abfcffea03aa8ddd3e927dd1a0beb64021a7fc1e276dbc" Oct 31 00:45:52.688425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.688400 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" event={"ID":"6a6c69c0-0b49-46e8-9573-5eac80d0f7a0","Type":"ContainerDied","Data":"171f4db493ba482b00f56b3b40fcaaeee60ed0c47b6f109f619f366b11acbbbc"} Oct 31 00:45:52.688542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.688525 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw" Oct 31 00:45:52.688682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.688670 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p" Oct 31 00:45:52.714399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.707472 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" Oct 31 00:45:52.714399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.707666 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq" event={"ID":"eb503a3b-5f90-4ba6-ad9d-9487b12a43df","Type":"ContainerDied","Data":"88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b"} Oct 31 00:45:52.714399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.707734 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f5d8ac94220aa0fee066100182156635ad931fc82127b0615ed7a4acb7d64b" Oct 31 00:45:52.732995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.730497 5011 scope.go:122] "RemoveContainer" containerID="9b39dac53c003be9ed4a2950c9e3b4670e687e68a2e6c6938bdd45d2c93321c7" Oct 31 00:45:52.761675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.761620 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:45:52.769792 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.769763 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f598cd05-867485b665-k6zsw"] Oct 31 00:45:52.777810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.777785 5011 patch_prober.go:29] interesting pod/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:45:52.777927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.777841 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 00:45:52.808296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.806645 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:45:52.815240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.814194 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f598cd05-6b5cf775f9-d2m9p"] Oct 31 00:45:52.915632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.915603 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:52.952031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952004 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952123 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952190 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952238 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952292 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.952374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952360 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952399 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.952537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952462 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952551 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952658 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\") pod \"267960c0-24ec-422e-bde3-4068242dff0c\" (UID: \"267960c0-24ec-422e-bde3-4068242dff0c\") " Oct 31 00:45:52.952735 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952689 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.952788 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952724 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:52.952926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.952866 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.953479 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.953455 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.953575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.953486 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.953575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.953502 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267960c0-24ec-422e-bde3-4068242dff0c-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.953575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.953518 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.953575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.953531 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:52.954943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.954915 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.955064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.955038 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.955064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.955059 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "kube-api-access-bhk8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.955594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:52.955571 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo" pod "267960c0-24ec-422e-bde3-4068242dff0c" (UID: "267960c0-24ec-422e-bde3-4068242dff0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:45:53.055011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.054975 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267960c0-24ec-422e-bde3-4068242dff0c-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:53.055011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.055011 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267960c0-24ec-422e-bde3-4068242dff0c-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:53.055220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.055025 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:53.055220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.055040 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhk8k\" (UniqueName: \"kubernetes.io/projected/267960c0-24ec-422e-bde3-4068242dff0c-kube-api-access-bhk8k\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:45:53.685906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.685872 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" path="/var/lib/kubelet/pods/6a6c69c0-0b49-46e8-9573-5eac80d0f7a0/volumes" Oct 31 00:45:53.686387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.686370 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987e4abc-a0c9-47f2-b3a5-b291b358cede" path="/var/lib/kubelet/pods/987e4abc-a0c9-47f2-b3a5-b291b358cede/volumes" Oct 31 00:45:53.723492 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.723459 5011 generic.go:356] "Generic (PLEG): container finished" podID="267960c0-24ec-422e-bde3-4068242dff0c" containerID="4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1" exitCode=0 Oct 31 00:45:53.723892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.723507 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" Oct 31 00:45:53.723892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.723539 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerDied","Data":"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1"} Oct 31 00:45:53.723892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.723596 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs" event={"ID":"267960c0-24ec-422e-bde3-4068242dff0c","Type":"ContainerDied","Data":"1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7"} Oct 31 00:45:53.723892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.723627 5011 scope.go:122] "RemoveContainer" containerID="4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1" Oct 31 00:45:53.753817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.753790 5011 scope.go:122] "RemoveContainer" containerID="3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351" Oct 31 00:45:53.761520 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:53.761488 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267960c0_24ec_422e_bde3_4068242dff0c.slice/crio-1688abf685b0490722f78b06afa17a8398008adda33f2a976558ddaba91407c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267960c0_24ec_422e_bde3_4068242dff0c.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:45:53.765843 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.765826 5011 scope.go:122] "RemoveContainer" containerID="4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1" Oct 31 00:45:53.766253 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:53.766215 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1\": container with ID starting with 4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1 not found: ID does not exist" containerID="4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1" Oct 31 00:45:53.766391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.766258 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1"} err="failed to get container status \"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1\": rpc error: code = NotFound desc = could not find container \"4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1\": container with ID starting with 4a21134b3c5286540407ab66d984dce9506693a60e9c9a9c2a70c708a3ba34a1 not found: ID does not exist" Oct 31 00:45:53.766391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.766283 5011 scope.go:122] "RemoveContainer" containerID="3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351" Oct 31 00:45:53.766600 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:45:53.766570 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351\": container with ID starting with 3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351 not found: ID does not exist" containerID="3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351" Oct 31 00:45:53.766669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.766608 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351"} err="failed to get container status \"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351\": rpc error: code = NotFound desc = could not find container \"3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351\": container with ID starting with 3795d30ffcfc7015895af277cf5ef60631887d5163b38d1cb8c49ebb73137351 not found: ID does not exist" Oct 31 00:45:53.790418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.790391 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:45:53.801099 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:53.801078 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a9461dc0-openshift-default-7d9bf49df-5xwzs"] Oct 31 00:45:54.061311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:54.061173 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn"] Oct 31 00:45:54.075296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:54.075264 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e828089-fjtsn"] Oct 31 00:45:55.689132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:55.689092 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267960c0-24ec-422e-bde3-4068242dff0c" path="/var/lib/kubelet/pods/267960c0-24ec-422e-bde3-4068242dff0c/volumes" Oct 31 00:45:55.689763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:45:55.689743 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad56d3cb-93a4-41ed-b2c5-906e936bd1e5" path="/var/lib/kubelet/pods/ad56d3cb-93a4-41ed-b2c5-906e936bd1e5/volumes" Oct 31 00:46:01.951309 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:46:01.951267 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:46:05.036127 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:05.035384 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c"] Oct 31 00:46:05.043172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:05.043129 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9c7fb04-bdw6c"] Oct 31 00:46:05.689509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:05.689472 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb50d93-ed63-4341-a950-3ce28473742b" path="/var/lib/kubelet/pods/bcb50d93-ed63-4341-a950-3ce28473742b/volumes" Oct 31 00:46:06.900253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:06.900191 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq"] Oct 31 00:46:06.914059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:06.914022 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b08f569a-mmwdq"] Oct 31 00:46:07.744260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:07.730543 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb503a3b-5f90-4ba6-ad9d-9487b12a43df" path="/var/lib/kubelet/pods/eb503a3b-5f90-4ba6-ad9d-9487b12a43df/volumes" Oct 31 00:46:24.154924 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:24.154883 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:46:31.399775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.399742 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j"] Oct 31 00:46:31.400370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400356 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eb503a3b-5f90-4ba6-ad9d-9487b12a43df" containerName="revoke-keys" Oct 31 00:46:31.400420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400371 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb503a3b-5f90-4ba6-ad9d-9487b12a43df" containerName="revoke-keys" Oct 31 00:46:31.400420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400389 5011 cpu_manager.go:414] "containerMap: removing container" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:46:31.400420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400394 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400427 5011 cpu_manager.go:414] "containerMap: removing container" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400434 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400444 5011 cpu_manager.go:414] "containerMap: removing container" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400449 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400456 5011 cpu_manager.go:414] "containerMap: removing container" podUID="987e4abc-a0c9-47f2-b3a5-b291b358cede" containerName="payload-pre-processing" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400461 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987e4abc-a0c9-47f2-b3a5-b291b358cede" containerName="payload-pre-processing" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400471 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" containerName="payload-processing" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400476 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" containerName="payload-processing" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400499 5011 cpu_manager.go:414] "containerMap: removing container" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:46:31.400515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400503 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ac7243-a0b2-49c6-b573-1e2217348a5d" containerName="cleanup" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400595 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb503a3b-5f90-4ba6-ad9d-9487b12a43df" containerName="revoke-keys" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400606 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="894678dc-069e-493d-8a3d-2b4f2a2b7d77" containerName="cleanup" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400613 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400621 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="267960c0-24ec-422e-bde3-4068242dff0c" containerName="istio-proxy" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400627 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="987e4abc-a0c9-47f2-b3a5-b291b358cede" containerName="payload-pre-processing" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400634 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193bf51-7fa2-4be3-9cd5-36ad5204bb73" containerName="cleanup" Oct 31 00:46:31.400809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.400641 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6c69c0-0b49-46e8-9573-5eac80d0f7a0" containerName="payload-processing" Oct 31 00:46:31.404689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.404673 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.426192 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.426163 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j"] Oct 31 00:46:31.429095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.429045 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:31.429340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.429261 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:46:31.488736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.488684 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g65x\" (UniqueName: \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.488929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.488910 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.489003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.488975 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.590671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.590640 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.590876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.590699 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.590983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.590958 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g65x\" (UniqueName: \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.591423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.591401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.593194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.593173 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.615839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.615804 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g65x\" (UniqueName: \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\") pod \"maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.719795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:31.719715 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:31.949061 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:46:31.949017 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:46:32.042422 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:46:32.042380 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283814c1_04c9_4678_b3b1_f433e334c49f.slice/crio-acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d WatchSource:0}: Error finding container acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d: Status 404 returned error can't find the container with id acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d Oct 31 00:46:32.042563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:32.042499 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j"] Oct 31 00:46:32.323988 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:32.323956 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" event={"ID":"283814c1-04c9-4678-b3b1-f433e334c49f","Type":"ContainerStarted","Data":"acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d"} Oct 31 00:46:33.335820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:33.335782 5011 generic.go:356] "Generic (PLEG): container finished" podID="283814c1-04c9-4678-b3b1-f433e334c49f" containerID="c3cac4fc29e005931cff475abd37a5b82b1b857b01c447746f228e51c44f7fbc" exitCode=0 Oct 31 00:46:33.336248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:33.335868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" event={"ID":"283814c1-04c9-4678-b3b1-f433e334c49f","Type":"ContainerDied","Data":"c3cac4fc29e005931cff475abd37a5b82b1b857b01c447746f228e51c44f7fbc"} Oct 31 00:46:34.510868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.510845 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:34.639916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.639830 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\") pod \"283814c1-04c9-4678-b3b1-f433e334c49f\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " Oct 31 00:46:34.639916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.639912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\" (UniqueName: \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\") pod \"283814c1-04c9-4678-b3b1-f433e334c49f\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " Oct 31 00:46:34.640153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.640050 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\") pod \"283814c1-04c9-4678-b3b1-f433e334c49f\" (UID: \"283814c1-04c9-4678-b3b1-f433e334c49f\") " Oct 31 00:46:34.640311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.640268 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca" pod "283814c1-04c9-4678-b3b1-f433e334c49f" (UID: "283814c1-04c9-4678-b3b1-f433e334c49f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:34.640815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.640790 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:46:34.642163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.642133 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca" pod "283814c1-04c9-4678-b3b1-f433e334c49f" (UID: "283814c1-04c9-4678-b3b1-f433e334c49f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:34.642378 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.642346 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x" pod "283814c1-04c9-4678-b3b1-f433e334c49f" (UID: "283814c1-04c9-4678-b3b1-f433e334c49f"). InnerVolumeSpecName "kube-api-access-2g65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:34.742445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.742413 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g65x\" (UniqueName: \"kubernetes.io/projected/283814c1-04c9-4678-b3b1-f433e334c49f-kube-api-access-2g65x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:46:34.742445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:34.742445 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283814c1-04c9-4678-b3b1-f433e334c49f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:46:35.355889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:35.355862 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" Oct 31 00:46:35.356172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:35.355891 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j" event={"ID":"283814c1-04c9-4678-b3b1-f433e334c49f","Type":"ContainerDied","Data":"acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d"} Oct 31 00:46:35.356328 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:35.356197 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acea7c68c89fdcc00f1c500d6cd5ca54c8665fd1c9c8ada4500c9cf3c3e6457d" Oct 31 00:46:40.694075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.694039 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 00:46:40.695018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.694995 5011 cpu_manager.go:414] "containerMap: removing container" podUID="283814c1-04c9-4678-b3b1-f433e334c49f" containerName="revoke-keys" Oct 31 00:46:40.695018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.695018 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283814c1-04c9-4678-b3b1-f433e334c49f" containerName="revoke-keys" Oct 31 00:46:40.695249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.695236 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="283814c1-04c9-4678-b3b1-f433e334c49f" containerName="revoke-keys" Oct 31 00:46:40.699958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.699932 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.733365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.733332 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 00:46:40.739996 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.739961 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.740243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.740197 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.740427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.740404 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv54p\" (UniqueName: \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.740572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.740556 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.806497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.806463 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-498580-serving-cert\"" Oct 31 00:46:40.842079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.842041 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.842371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.842350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.842510 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:46:40.842490 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-serving-cert: secret "maas-api-e2e-shared-a-w0-498580-serving-cert" not found Oct 31 00:46:40.842510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.842502 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.842640 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.842526 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv54p\" (UniqueName: \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.842640 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:46:40.842562 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls podName:b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc nodeName:}" failed. No retries permitted until 2026-10-31 00:46:41.342541426 +0000 UTC m=+1669.791629044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls") pod "maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" (UID: "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc") : secret "maas-api-e2e-shared-a-w0-498580-serving-cert" not found Oct 31 00:46:40.842864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.842835 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.845290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.845261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:40.864394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:40.864367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv54p\" (UniqueName: \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:41.354617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:41.354561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:41.357666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:41.357635 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:41.632512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:41.632415 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:42.207845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:42.207807 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 00:46:42.472742 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:42.470541 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" event={"ID":"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc","Type":"ContainerStarted","Data":"73ed46ca9f59f88185a249a3736b0c42406acebd18e102cf150f3c7e101a5e15"} Oct 31 00:46:43.492261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:43.492222 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" event={"ID":"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc","Type":"ContainerStarted","Data":"a967e5a191830536818391a602225637df24ad546fe20efa3fd7759fb29e122b"} Oct 31 00:46:43.493650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:43.493617 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:43.557791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:43.557737 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" podStartSLOduration=3.198152884 podStartE2EDuration="3.557722446s" totalImagesPullingTime="359.569562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:43.555282251 +0000 UTC m=+1672.004369890" watchObservedRunningTime="2026-10-31 00:46:43.557722446 +0000 UTC m=+1672.006810229" Oct 31 00:46:46.433286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:46.433240 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j"] Oct 31 00:46:46.444582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:46.444546 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2cd8283-gpl4j"] Oct 31 00:46:47.685912 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:47.685876 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283814c1-04c9-4678-b3b1-f433e334c49f" path="/var/lib/kubelet/pods/283814c1-04c9-4678-b3b1-f433e334c49f/volumes" Oct 31 00:46:50.532728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:50.532700 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 00:46:52.067416 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.067380 5011 scope.go:122] "RemoveContainer" containerID="251430c14a07cc2e07a43059c05833493b19413cf95abeb68e48086fc4d71ae7" Oct 31 00:46:52.078699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.078674 5011 scope.go:122] "RemoveContainer" containerID="a843fbca1a9e9f392383e4bc169d25f36c07667e366da88b47b1b7a08dce548e" Oct 31 00:46:52.088349 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.088324 5011 scope.go:122] "RemoveContainer" containerID="5d73501634ed93ab315b23154800b2227e3ca0ed8829a73a9ec32dd2fccca369" Oct 31 00:46:52.100109 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.100089 5011 scope.go:122] "RemoveContainer" containerID="e3ba72afc6af3c8919477e07b914a8ad4b65f36a2f50d77e143c4e1cb3430277" Oct 31 00:46:52.110076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.110057 5011 scope.go:122] "RemoveContainer" containerID="1d87347caba2c3ca267e1349b39d3549279f2723ec7488a5ea37408f9908f935" Oct 31 00:46:52.119790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.119770 5011 scope.go:122] "RemoveContainer" containerID="2047ac0c73c233acbdd4e97d71718a84b56d06329a4837bc7e486569a6b53609" Oct 31 00:46:52.129874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.129854 5011 scope.go:122] "RemoveContainer" containerID="8bd7165a8e228df9ef55b93e282b0bec27bb485c966a3e9d369c64026c1ad9b7" Oct 31 00:46:52.139417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.139400 5011 scope.go:122] "RemoveContainer" containerID="df6fb815097ccd5f75fe56ce0d8315f3cf181d4b7d66df414eaa76de39861314" Oct 31 00:46:52.148573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.148553 5011 scope.go:122] "RemoveContainer" containerID="8411b536bb2fbceffe636fdc81b1bacb442ff11d8a98af60cc16136460247786" Oct 31 00:46:52.158683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:46:52.158656 5011 scope.go:122] "RemoveContainer" containerID="48f56a315d802faa53d3055ed2d205b262a68a44e71c2cc07031ec058aeb595c" Oct 31 00:47:01.819114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:01.819069 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:47:01.953020 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:01.952970 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:47:07.612787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.612747 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 00:47:07.618347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.618321 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.640393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.640350 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3c90d8-openshift-default-dockercfg-vpjjh\"" Oct 31 00:47:07.659187 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659155 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 00:47:07.659567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659513 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.659659 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659589 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhkb\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.659736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659716 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.659836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659803 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.660023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.659978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.660267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.660024 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.660267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.660073 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.660267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.660139 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.660267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.660167 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.845244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.844990 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.845244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.845116 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.845244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.845144 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.845244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.845192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.849265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.847464 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.849265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.847706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.866406 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.866455 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.866656 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.866701 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhkb\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.866862 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.867348 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.868408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.867983 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.878864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.874039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.878864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.874800 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.878864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.875477 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.890880 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.887694 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.901631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.901566 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhkb\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:07.945156 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:07.945119 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:08.338052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.338023 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 00:47:08.339957 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:08.339921 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d640c7_caa1_4ec8_ba7a_d66bf55a69b0.slice/crio-1bc5b1cc35fc440e6c717c280691c8c08d7bc59499a2a58a3d36c229d21a56e2 WatchSource:0}: Error finding container 1bc5b1cc35fc440e6c717c280691c8c08d7bc59499a2a58a3d36c229d21a56e2: Status 404 returned error can't find the container with id 1bc5b1cc35fc440e6c717c280691c8c08d7bc59499a2a58a3d36c229d21a56e2 Oct 31 00:47:08.343538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.343498 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:08.343628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.343591 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:08.343668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.343635 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:08.893294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.893244 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerStarted","Data":"43072c510fb38961ff94e7589fb2210cb2a217a74074c7b95fda947da511a5e3"} Oct 31 00:47:08.893704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.893325 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerStarted","Data":"1bc5b1cc35fc440e6c717c280691c8c08d7bc59499a2a58a3d36c229d21a56e2"} Oct 31 00:47:08.945971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.945932 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:08.947000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:08.946956 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podStartSLOduration=1.9469411989999998 podStartE2EDuration="1.946941199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:08.940519716 +0000 UTC m=+1697.389607356" watchObservedRunningTime="2026-10-31 00:47:08.946941199 +0000 UTC m=+1697.396028837" Oct 31 00:47:09.078156 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.078115 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:47:09.083383 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.083353 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.099643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.099612 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-208e9f32-serving-cert\"" Oct 31 00:47:09.124743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.124319 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.124743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.124388 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.124743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.124415 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vn5x\" (UniqueName: \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.124743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.124486 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.147065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.146981 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:47:09.226487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.226453 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.226687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.226525 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.226687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.226557 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vn5x\" (UniqueName: \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.226687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.226637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.227473 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:09.227449 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-serving-cert: secret "maas-api-e2e-mt-208e9f32-serving-cert" not found Oct 31 00:47:09.227986 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:09.227967 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls podName:b8d567ec-9a8c-458a-91b9-b7c61fb8ca60 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:09.727945007 +0000 UTC m=+1698.177032646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls") pod "maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60") : secret "maas-api-e2e-mt-208e9f32-serving-cert" not found Oct 31 00:47:09.229302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.228475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.265229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.259953 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.265229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.264933 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vn5x\" (UniqueName: \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.741021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.740976 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:09.741265 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:09.741094 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-serving-cert: secret "maas-api-e2e-mt-208e9f32-serving-cert" not found Oct 31 00:47:09.741265 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:09.741166 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls podName:b8d567ec-9a8c-458a-91b9-b7c61fb8ca60 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:10.741147176 +0000 UTC m=+1699.190234794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls") pod "maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60") : secret "maas-api-e2e-mt-208e9f32-serving-cert" not found Oct 31 00:47:09.953278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:09.953244 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:10.759931 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:10.759882 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:10.762523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:10.762488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:10.913930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:10.913897 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:10.915262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:10.915202 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:47:10.926980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:10.926952 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:11.278714 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:11.278683 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:47:11.279100 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:11.279041 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d567ec_9a8c_458a_91b9_b7c61fb8ca60.slice/crio-6bb42e7300db9f44533181b2130586433a6f1781e80d0d0131491652f8fe68f2 WatchSource:0}: Error finding container 6bb42e7300db9f44533181b2130586433a6f1781e80d0d0131491652f8fe68f2: Status 404 returned error can't find the container with id 6bb42e7300db9f44533181b2130586433a6f1781e80d0d0131491652f8fe68f2 Oct 31 00:47:11.952028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:11.946832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" event={"ID":"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60","Type":"ContainerStarted","Data":"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5"} Oct 31 00:47:11.952028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:11.946896 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" event={"ID":"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60","Type":"ContainerStarted","Data":"6bb42e7300db9f44533181b2130586433a6f1781e80d0d0131491652f8fe68f2"} Oct 31 00:47:11.952028 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:11.946945 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:11.991252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:11.991144 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" podStartSLOduration=2.634335716 podStartE2EDuration="2.991121564s" totalImagesPullingTime="356.785848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:11.986809605 +0000 UTC m=+1700.435897243" watchObservedRunningTime="2026-10-31 00:47:11.991121564 +0000 UTC m=+1700.440209205" Oct 31 00:47:18.969267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:18.969237 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:47:31.948304 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:31.948269 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:47:44.569505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.569468 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 00:47:44.575232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.575191 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.595782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.595758 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-bcdb4e\"" Oct 31 00:47:44.601263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.598646 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 00:47:44.698149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.694222 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.698149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.694562 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z88\" (UniqueName: \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.804228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.803504 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.804228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.803900 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z88\" (UniqueName: \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.845273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.837564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:44.915119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:44.915050 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z88\" (UniqueName: \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\") pod \"payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:45.211883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.211799 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:45.273353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.273315 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:47:45.283842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.283812 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.319116 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.317056 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f2bdf3-openshift-default-dockercfg-xzqfh\"" Oct 31 00:47:45.326303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.325582 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:47:45.379710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.379679 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcww\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.379883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.379733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.379883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.379828 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.379889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.379991 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.380108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.380272 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.380398 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.380502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.380443 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481535 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481614 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481594 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcww\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481734 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481807 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.481970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482005 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481963 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482042 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.481994 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.482236 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.482391 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.482480 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482741 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.482722 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.482862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.482842 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.484171 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.484150 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.484503 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.484482 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.505642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.505618 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.505739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.505667 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcww\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.587776 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:45.587747 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39ac115_645a_4611_8d44_d4aea5b4b13c.slice/crio-d7afa62510156e80e5b6c751211ba9cde09651a58bab89f8ea0573cfcbf7af18 WatchSource:0}: Error finding container d7afa62510156e80e5b6c751211ba9cde09651a58bab89f8ea0573cfcbf7af18: Status 404 returned error can't find the container with id d7afa62510156e80e5b6c751211ba9cde09651a58bab89f8ea0573cfcbf7af18 Oct 31 00:47:45.588121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.587993 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 00:47:45.617627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.617602 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:45.955842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.955809 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:47:45.957660 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:45.957621 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecdd291_ba26_4816_aab6_9a903087606b.slice/crio-c8a0c802fc35e42c3e083a6a86b1bbeec65dfd4fb2c499d17f9c07ed95283e48 WatchSource:0}: Error finding container c8a0c802fc35e42c3e083a6a86b1bbeec65dfd4fb2c499d17f9c07ed95283e48: Status 404 returned error can't find the container with id c8a0c802fc35e42c3e083a6a86b1bbeec65dfd4fb2c499d17f9c07ed95283e48 Oct 31 00:47:45.960855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.960811 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:45.960975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.960904 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:45.960975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:45.960954 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:46.336896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.336858 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" event={"ID":"f39ac115-645a-4611-8d44-d4aea5b4b13c","Type":"ContainerStarted","Data":"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64"} Oct 31 00:47:46.337164 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.337148 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" event={"ID":"f39ac115-645a-4611-8d44-d4aea5b4b13c","Type":"ContainerStarted","Data":"d7afa62510156e80e5b6c751211ba9cde09651a58bab89f8ea0573cfcbf7af18"} Oct 31 00:47:46.338490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.338465 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:46.342279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.342243 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" event={"ID":"3ecdd291-ba26-4816-aab6-9a903087606b","Type":"ContainerStarted","Data":"6e40c587d4cb609d65b2a961eaf05499f550b3876bbac8f70f48c660cd17009a"} Oct 31 00:47:46.342456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.342442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" event={"ID":"3ecdd291-ba26-4816-aab6-9a903087606b","Type":"ContainerStarted","Data":"c8a0c802fc35e42c3e083a6a86b1bbeec65dfd4fb2c499d17f9c07ed95283e48"} Oct 31 00:47:46.381138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.381066 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" podStartSLOduration=2.381046955 podStartE2EDuration="2.381046955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:46.375169077 +0000 UTC m=+1734.824256717" watchObservedRunningTime="2026-10-31 00:47:46.381046955 +0000 UTC m=+1734.830134597" Oct 31 00:47:46.418317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.418263 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podStartSLOduration=1.4182447360000001 podStartE2EDuration="1.418244736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:46.416540085 +0000 UTC m=+1734.865627724" watchObservedRunningTime="2026-10-31 00:47:46.418244736 +0000 UTC m=+1734.867332376" Oct 31 00:47:46.620402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:46.620298 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:47.623183 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:47.623146 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:47:47.623675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:47.623244 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:47:48.477586 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.477549 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-4fb6bf"] Oct 31 00:47:48.482267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.482248 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:48.500141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.500118 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-4fb6bf"] Oct 31 00:47:48.503771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.503747 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\"" Oct 31 00:47:48.564198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.564156 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zf7\" (UniqueName: \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\") pod \"test-curl-10893-4fb6bf\" (UID: \"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9\") " pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:48.642232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.641547 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:47:48.642232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.641607 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:47:48.666067 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.665755 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zf7\" (UniqueName: \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\") pod \"test-curl-10893-4fb6bf\" (UID: \"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9\") " pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:48.687602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.687570 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zf7\" (UniqueName: \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\") pod \"test-curl-10893-4fb6bf\" (UID: \"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9\") " pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:48.795151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:48.795073 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:49.169500 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.163300 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc"] Oct 31 00:47:49.173348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.173267 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-4fb6bf"] Oct 31 00:47:49.173476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.173403 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.210104 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.210071 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:49.210293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.210281 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:47:49.211242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.211164 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc"] Oct 31 00:47:49.286253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.285635 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.286253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.285779 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.286253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.285912 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qtm\" (UniqueName: \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.390759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.387674 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.390759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.387840 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.390759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.387978 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qtm\" (UniqueName: \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.390759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.389156 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.396150 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.396060 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.404469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.404444 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-4fb6bf" event={"ID":"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9","Type":"ContainerStarted","Data":"7fe4245a4ad29cb08608c3ce245b1afb96013ee4b7aa1d921d4fecb81922c024"} Oct 31 00:47:49.411098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.411044 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qtm\" (UniqueName: \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\") pod \"maas-api-revoke-sub-06a5ebf6-6q4xc\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.592438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.592409 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:49.622250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.619535 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:47:49.622250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.619596 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:47:49.956400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:49.956369 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc"] Oct 31 00:47:50.413894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:50.413864 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" event={"ID":"5a696096-a2b1-4168-8f3d-f92c9e82831c","Type":"ContainerStarted","Data":"ae69589a4c157662fd09c3933b7ae810fc417b4fce70125644ad60bba9eed516"} Oct 31 00:47:50.415554 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:50.415528 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-4fb6bf" event={"ID":"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9","Type":"ContainerStarted","Data":"d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308"} Oct 31 00:47:50.462026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:50.461977 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-4fb6bf" podStartSLOduration=2.46196327 podStartE2EDuration="2.46196327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:50.458112553 +0000 UTC m=+1738.907200194" watchObservedRunningTime="2026-10-31 00:47:50.46196327 +0000 UTC m=+1738.911050909" Oct 31 00:47:50.644229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:50.642444 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:47:50.644229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:50.642500 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:47:51.451161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.451062 5011 generic.go:356] "Generic (PLEG): container finished" podID="5a696096-a2b1-4168-8f3d-f92c9e82831c" containerID="08e5322f1fbb787971d77fc46bb3cd4b79dba1bb003426786fe62ba584c85cb4" exitCode=0 Oct 31 00:47:51.451658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.451490 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" event={"ID":"5a696096-a2b1-4168-8f3d-f92c9e82831c","Type":"ContainerDied","Data":"08e5322f1fbb787971d77fc46bb3cd4b79dba1bb003426786fe62ba584c85cb4"} Oct 31 00:47:51.506230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.506175 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-4fb6bf"] Oct 31 00:47:51.516253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.515889 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-4fb6bf"] Oct 31 00:47:51.652234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.651527 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 00:47:51.652234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.651578 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 00:47:51.692815 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.692786 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-e950be"] Oct 31 00:47:51.693487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.693470 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" containerName="test-curl-10893-4fb6bf" Oct 31 00:47:51.693538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.693489 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" containerName="test-curl-10893-4fb6bf" Oct 31 00:47:51.693602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.693593 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" containerName="test-curl-10893-4fb6bf" Oct 31 00:47:51.697296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.697277 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:51.759295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.759259 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-e950be"] Oct 31 00:47:51.767435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.766843 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvf8j\" (UniqueName: \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\") pod \"test-curl-10893-e950be\" (UID: \"688ecf3c-9fec-4376-a44f-c4089761b76a\") " pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:51.870794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.870759 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvf8j\" (UniqueName: \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\") pod \"test-curl-10893-e950be\" (UID: \"688ecf3c-9fec-4376-a44f-c4089761b76a\") " pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:51.902239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:51.901868 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvf8j\" (UniqueName: \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\") pod \"test-curl-10893-e950be\" (UID: \"688ecf3c-9fec-4376-a44f-c4089761b76a\") " pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:52.010826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.010747 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:52.073609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.073530 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt"] Oct 31 00:47:52.082884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.082847 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851dae1f-k48tt"] Oct 31 00:47:52.273654 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.273567 5011 scope.go:122] "RemoveContainer" containerID="4622f5d5cae2eb3432619914e06a9d98347505b70f7f0625da98e33ae00e85d6" Oct 31 00:47:52.356443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.356411 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-e950be"] Oct 31 00:47:52.357660 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:52.357631 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688ecf3c_9fec_4376_a44f_c4089761b76a.slice/crio-88e320eb7799cde12177a9afcc9e7f14869e6c13a3c457fe898a95d3fb18b306 WatchSource:0}: Error finding container 88e320eb7799cde12177a9afcc9e7f14869e6c13a3c457fe898a95d3fb18b306: Status 404 returned error can't find the container with id 88e320eb7799cde12177a9afcc9e7f14869e6c13a3c457fe898a95d3fb18b306 Oct 31 00:47:52.463151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.463111 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-e950be" event={"ID":"688ecf3c-9fec-4376-a44f-c4089761b76a","Type":"ContainerStarted","Data":"d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e"} Oct 31 00:47:52.463862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.463167 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-e950be" event={"ID":"688ecf3c-9fec-4376-a44f-c4089761b76a","Type":"ContainerStarted","Data":"88e320eb7799cde12177a9afcc9e7f14869e6c13a3c457fe898a95d3fb18b306"} Oct 31 00:47:52.463862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.463610 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-4fb6bf" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" containerName="test-curl-10893-4fb6bf" containerID="cri-o://d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308" gracePeriod=2 Oct 31 00:47:52.500136 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.500076 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-e950be" podStartSLOduration=1.500057135 podStartE2EDuration="1.500057135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:52.498485659 +0000 UTC m=+1740.947573300" watchObservedRunningTime="2026-10-31 00:47:52.500057135 +0000 UTC m=+1740.949144775" Oct 31 00:47:52.629927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.629893 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:52.630719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.630695 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:52.631772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.631749 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:47:52.657888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.657863 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:52.787106 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.787017 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\") pod \"5a696096-a2b1-4168-8f3d-f92c9e82831c\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " Oct 31 00:47:52.787334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.787136 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\" (UniqueName: \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\") pod \"5a696096-a2b1-4168-8f3d-f92c9e82831c\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " Oct 31 00:47:52.787415 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.787353 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\") pod \"5a696096-a2b1-4168-8f3d-f92c9e82831c\" (UID: \"5a696096-a2b1-4168-8f3d-f92c9e82831c\") " Oct 31 00:47:52.794464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.794430 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca" pod "5a696096-a2b1-4168-8f3d-f92c9e82831c" (UID: "5a696096-a2b1-4168-8f3d-f92c9e82831c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:52.797883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.797853 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm" pod "5a696096-a2b1-4168-8f3d-f92c9e82831c" (UID: "5a696096-a2b1-4168-8f3d-f92c9e82831c"). InnerVolumeSpecName "kube-api-access-f4qtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:52.799041 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.798411 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca" pod "5a696096-a2b1-4168-8f3d-f92c9e82831c" (UID: "5a696096-a2b1-4168-8f3d-f92c9e82831c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:52.890047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.890018 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:52.890047 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.890048 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a696096-a2b1-4168-8f3d-f92c9e82831c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:52.890251 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:52.890064 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qtm\" (UniqueName: \"kubernetes.io/projected/5a696096-a2b1-4168-8f3d-f92c9e82831c-kube-api-access-f4qtm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:53.472732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.472702 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" Oct 31 00:47:53.473139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.472739 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc" event={"ID":"5a696096-a2b1-4168-8f3d-f92c9e82831c","Type":"ContainerDied","Data":"ae69589a4c157662fd09c3933b7ae810fc417b4fce70125644ad60bba9eed516"} Oct 31 00:47:53.473139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.472789 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae69589a4c157662fd09c3933b7ae810fc417b4fce70125644ad60bba9eed516" Oct 31 00:47:53.517170 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.517126 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-e950be"] Oct 31 00:47:53.524284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.524255 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-e950be"] Oct 31 00:47:53.685055 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.685021 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9df0724-7a93-42a6-94f4-451e184f5ff3" path="/var/lib/kubelet/pods/c9df0724-7a93-42a6-94f4-451e184f5ff3/volumes" Oct 31 00:47:53.705576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.705528 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-c5df43"] Oct 31 00:47:53.706223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706195 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5a696096-a2b1-4168-8f3d-f92c9e82831c" containerName="revoke-keys" Oct 31 00:47:53.706289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706223 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a696096-a2b1-4168-8f3d-f92c9e82831c" containerName="revoke-keys" Oct 31 00:47:53.706289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706243 5011 cpu_manager.go:414] "containerMap: removing container" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" containerName="test-curl-10893-e950be" Oct 31 00:47:53.706289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706249 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" containerName="test-curl-10893-e950be" Oct 31 00:47:53.706393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706353 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" containerName="test-curl-10893-e950be" Oct 31 00:47:53.706393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.706370 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a696096-a2b1-4168-8f3d-f92c9e82831c" containerName="revoke-keys" Oct 31 00:47:53.711003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.710984 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:53.727695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.727643 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-c5df43"] Oct 31 00:47:53.804294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.804254 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstxz\" (UniqueName: \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\") pod \"test-curl-10893-c5df43\" (UID: \"216f44fd-06d8-4fd5-94fb-b997f430dbb4\") " pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:53.905870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.905838 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstxz\" (UniqueName: \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\") pod \"test-curl-10893-c5df43\" (UID: \"216f44fd-06d8-4fd5-94fb-b997f430dbb4\") " pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:53.928579 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:53.928549 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstxz\" (UniqueName: \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\") pod \"test-curl-10893-c5df43\" (UID: \"216f44fd-06d8-4fd5-94fb-b997f430dbb4\") " pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:54.024774 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.024698 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:54.334694 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:54.334664 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod216f44fd_06d8_4fd5_94fb_b997f430dbb4.slice/crio-717c05f5fa5cf2fba2464f25a9b9b9e9817c23d2ab6ca9656e9f98fa603c490e WatchSource:0}: Error finding container 717c05f5fa5cf2fba2464f25a9b9b9e9817c23d2ab6ca9656e9f98fa603c490e: Status 404 returned error can't find the container with id 717c05f5fa5cf2fba2464f25a9b9b9e9817c23d2ab6ca9656e9f98fa603c490e Oct 31 00:47:54.335177 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.335152 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-c5df43"] Oct 31 00:47:54.487602 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.487567 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-e950be" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" containerName="test-curl-10893-e950be" containerID="cri-o://d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e" gracePeriod=2 Oct 31 00:47:54.489173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.487726 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-c5df43" event={"ID":"216f44fd-06d8-4fd5-94fb-b997f430dbb4","Type":"ContainerStarted","Data":"595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732"} Oct 31 00:47:54.489173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.487795 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-c5df43" event={"ID":"216f44fd-06d8-4fd5-94fb-b997f430dbb4","Type":"ContainerStarted","Data":"717c05f5fa5cf2fba2464f25a9b9b9e9817c23d2ab6ca9656e9f98fa603c490e"} Oct 31 00:47:54.525708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.525358 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-c5df43" podStartSLOduration=1.525339422 podStartE2EDuration="1.525339422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:54.523550129 +0000 UTC m=+1742.972637768" watchObservedRunningTime="2026-10-31 00:47:54.525339422 +0000 UTC m=+1742.974427079" Oct 31 00:47:54.707812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.707787 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:54.820433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.820401 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\" (UniqueName: \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\") pod \"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9\" (UID: \"1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9\") " Oct 31 00:47:54.822779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.822747 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7" pod "1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" (UID: "1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9"). InnerVolumeSpecName "kube-api-access-p5zf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:54.922954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:54.922875 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zf7\" (UniqueName: \"kubernetes.io/projected/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9-kube-api-access-p5zf7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:55.482948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.482910 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-c5df43"] Oct 31 00:47:55.488445 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.488413 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-c5df43"] Oct 31 00:47:55.518433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.518393 5011 generic.go:356] "Generic (PLEG): container finished" podID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" containerID="d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308" exitCode=137 Oct 31 00:47:55.520082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.520053 5011 scope.go:122] "RemoveContainer" containerID="d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308" Oct 31 00:47:55.526082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.520058 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:55.546190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.546166 5011 scope.go:122] "RemoveContainer" containerID="d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308" Oct 31 00:47:55.547124 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:55.546586 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308\": container with ID starting with d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308 not found: ID does not exist" containerID="d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308" Oct 31 00:47:55.547124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.546629 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308"} err="failed to get container status \"d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308\": rpc error: code = NotFound desc = could not find container \"d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308\": container with ID starting with d5f039dd11280048d6533816fc3f26363c9d0cf3812cbbdbd08e0a5246146308 not found: ID does not exist" Oct 31 00:47:55.558325 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:55.558291 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-4fb6bf\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:55.653969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.653931 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10893-f1f0f8"] Oct 31 00:47:55.654721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.654699 5011 cpu_manager.go:414] "containerMap: removing container" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" containerName="test-curl-10893-c5df43" Oct 31 00:47:55.654721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.654720 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" containerName="test-curl-10893-c5df43" Oct 31 00:47:55.654886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.654862 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" containerName="test-curl-10893-c5df43" Oct 31 00:47:55.659872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.659846 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:47:55.685856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.685820 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" path="/var/lib/kubelet/pods/1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9/volumes" Oct 31 00:47:55.686221 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.686185 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-f1f0f8"] Oct 31 00:47:55.705997 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:55.705965 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-4fb6bf\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1ffd84d0-e6cd-42f5-8e56-2ff6e7fef4e9" pod="opendatahub/test-curl-10893-4fb6bf" Oct 31 00:47:55.735827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.735758 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgm9v\" (UniqueName: \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\") pod \"test-curl-10893-f1f0f8\" (UID: \"71e3a1f4-081a-47c1-9cbe-1b674b8aa417\") " pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:47:55.838026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.837989 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgm9v\" (UniqueName: \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\") pod \"test-curl-10893-f1f0f8\" (UID: \"71e3a1f4-081a-47c1-9cbe-1b674b8aa417\") " pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:47:55.860924 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.860891 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgm9v\" (UniqueName: \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\") pod \"test-curl-10893-f1f0f8\" (UID: \"71e3a1f4-081a-47c1-9cbe-1b674b8aa417\") " pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:47:55.975178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:55.975141 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:47:56.290664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.290635 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10893-f1f0f8"] Oct 31 00:47:56.291441 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:56.291409 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e3a1f4_081a_47c1_9cbe_1b674b8aa417.slice/crio-f1a2b6f0bd88125bb7bd5eb6aa84be6df33ed02090a3520452fb2853214f2784 WatchSource:0}: Error finding container f1a2b6f0bd88125bb7bd5eb6aa84be6df33ed02090a3520452fb2853214f2784: Status 404 returned error can't find the container with id f1a2b6f0bd88125bb7bd5eb6aa84be6df33ed02090a3520452fb2853214f2784 Oct 31 00:47:56.531783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.531744 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-f1f0f8" event={"ID":"71e3a1f4-081a-47c1-9cbe-1b674b8aa417","Type":"ContainerStarted","Data":"efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa"} Oct 31 00:47:56.532247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.531795 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10893-f1f0f8" event={"ID":"71e3a1f4-081a-47c1-9cbe-1b674b8aa417","Type":"ContainerStarted","Data":"f1a2b6f0bd88125bb7bd5eb6aa84be6df33ed02090a3520452fb2853214f2784"} Oct 31 00:47:56.533412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.533382 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-c5df43" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" containerName="test-curl-10893-c5df43" containerID="cri-o://595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732" gracePeriod=2 Oct 31 00:47:56.580485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.580426 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10893-f1f0f8" podStartSLOduration=1.580410336 podStartE2EDuration="1.580410336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:56.575834244 +0000 UTC m=+1745.024921885" watchObservedRunningTime="2026-10-31 00:47:56.580410336 +0000 UTC m=+1745.029497977" Oct 31 00:47:56.717130 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.717104 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:56.858709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.858675 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\" (UniqueName: \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\") pod \"688ecf3c-9fec-4376-a44f-c4089761b76a\" (UID: \"688ecf3c-9fec-4376-a44f-c4089761b76a\") " Oct 31 00:47:56.860861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.860830 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j" pod "688ecf3c-9fec-4376-a44f-c4089761b76a" (UID: "688ecf3c-9fec-4376-a44f-c4089761b76a"). InnerVolumeSpecName "kube-api-access-wvf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:56.960850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:56.960821 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvf8j\" (UniqueName: \"kubernetes.io/projected/688ecf3c-9fec-4376-a44f-c4089761b76a-kube-api-access-wvf8j\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:57.544745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.544710 5011 generic.go:356] "Generic (PLEG): container finished" podID="688ecf3c-9fec-4376-a44f-c4089761b76a" containerID="d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e" exitCode=137 Oct 31 00:47:57.545202 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.544764 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:57.545202 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.544803 5011 scope.go:122] "RemoveContainer" containerID="d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e" Oct 31 00:47:57.557634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.557559 5011 scope.go:122] "RemoveContainer" containerID="d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e" Oct 31 00:47:57.558017 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:57.557980 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e\": container with ID starting with d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e not found: ID does not exist" containerID="d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e" Oct 31 00:47:57.558119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.558021 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e"} err="failed to get container status \"d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e\": rpc error: code = NotFound desc = could not find container \"d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e\": container with ID starting with d78705578bf05feb74ce1768713a98f56460c59fb763d5948c4d35f39b56a04e not found: ID does not exist" Oct 31 00:47:57.564911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.564885 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10893-f1f0f8"] Oct 31 00:47:57.570238 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:57.570178 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-e950be\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" pod="opendatahub/test-curl-10893-e950be" Oct 31 00:47:57.570829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.570806 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10893-f1f0f8"] Oct 31 00:47:57.684904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:57.684871 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688ecf3c-9fec-4376-a44f-c4089761b76a" path="/var/lib/kubelet/pods/688ecf3c-9fec-4376-a44f-c4089761b76a/volumes" Oct 31 00:47:58.378755 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.378727 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 00:47:58.558071 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.558033 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10893-f1f0f8" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" containerName="test-curl-10893-f1f0f8" containerID="cri-o://efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa" gracePeriod=2 Oct 31 00:47:58.626577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.626445 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:47:58.628066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.628038 5011 cpu_manager.go:414] "containerMap: removing container" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" containerName="test-curl-10893-f1f0f8" Oct 31 00:47:58.628270 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.628255 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" containerName="test-curl-10893-f1f0f8" Oct 31 00:47:58.628620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.628602 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" containerName="test-curl-10893-f1f0f8" Oct 31 00:47:58.634150 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.634105 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.670713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.670638 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-14e039-openshift-default-dockercfg-cwjf5\"" Oct 31 00:47:58.687094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.687065 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797626 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797755 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797832 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797873 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797930 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797962 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.797987 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.798241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2q7k\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:58.809699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:58.798300 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.009565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.009450 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.009872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.009849 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010023 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010170 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010156 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010336 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010321 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010443 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010665 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.010975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.010955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2q7k\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.011124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.011110 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.011961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.011902 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.012799 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.012778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.023470 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.023292 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.043228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.023731 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.043520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.023993 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.054585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.054553 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.072069 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.072033 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2q7k\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.084736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.084702 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.103376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.103326 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\") pod \"e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.173522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.173492 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:59.240261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.237500 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\" (UniqueName: \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\") pod \"216f44fd-06d8-4fd5-94fb-b997f430dbb4\" (UID: \"216f44fd-06d8-4fd5-94fb-b997f430dbb4\") " Oct 31 00:47:59.255951 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.255910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz" pod "216f44fd-06d8-4fd5-94fb-b997f430dbb4" (UID: "216f44fd-06d8-4fd5-94fb-b997f430dbb4"). InnerVolumeSpecName "kube-api-access-jstxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:59.284615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.284508 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:47:59.340056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.340015 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jstxz\" (UniqueName: \"kubernetes.io/projected/216f44fd-06d8-4fd5-94fb-b997f430dbb4-kube-api-access-jstxz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:47:59.569585 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.569558 5011 generic.go:356] "Generic (PLEG): container finished" podID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" containerID="595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732" exitCode=137 Oct 31 00:47:59.569969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.569615 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:59.569969 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.569628 5011 scope.go:122] "RemoveContainer" containerID="595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732" Oct 31 00:47:59.584294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.584270 5011 scope.go:122] "RemoveContainer" containerID="595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732" Oct 31 00:47:59.584622 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:59.584600 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732\": container with ID starting with 595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732 not found: ID does not exist" containerID="595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732" Oct 31 00:47:59.584708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.584633 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732"} err="failed to get container status \"595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732\": rpc error: code = NotFound desc = could not find container \"595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732\": container with ID starting with 595f3d2415ebf5e2c19dd663c1d3ad039264b4d9eae267ca3e83fb1534cad732 not found: ID does not exist" Oct 31 00:47:59.591860 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:47:59.591832 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-c5df43\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" pod="opendatahub/test-curl-10893-c5df43" Oct 31 00:47:59.656291 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:47:59.656067 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3040c175_a485_4d7a_9b8b_60bcfa6340dd.slice/crio-e45df68407e4b1ca9ac6d7a8dfd14d9db477ff7e74ae33ddfae2b21ccf358e56 WatchSource:0}: Error finding container e45df68407e4b1ca9ac6d7a8dfd14d9db477ff7e74ae33ddfae2b21ccf358e56: Status 404 returned error can't find the container with id e45df68407e4b1ca9ac6d7a8dfd14d9db477ff7e74ae33ddfae2b21ccf358e56 Oct 31 00:47:59.659230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.659178 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:59.659343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.659285 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:59.659404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.659346 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:47:59.689379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.689329 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f44fd-06d8-4fd5-94fb-b997f430dbb4" path="/var/lib/kubelet/pods/216f44fd-06d8-4fd5-94fb-b997f430dbb4/volumes" Oct 31 00:47:59.690245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.690224 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:47:59.811932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:47:59.811836 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:48:00.584631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:00.583639 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerStarted","Data":"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7"} Oct 31 00:48:00.584631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:00.583698 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerStarted","Data":"e45df68407e4b1ca9ac6d7a8dfd14d9db477ff7e74ae33ddfae2b21ccf358e56"} Oct 31 00:48:00.730226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:00.726455 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podStartSLOduration=2.726430248 podStartE2EDuration="2.726430248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:00.651872113 +0000 UTC m=+1749.100959755" watchObservedRunningTime="2026-10-31 00:48:00.726430248 +0000 UTC m=+1749.175517888" Oct 31 00:48:01.152179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.151960 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:48:01.234742 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.234646 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\" (UniqueName: \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\") pod \"71e3a1f4-081a-47c1-9cbe-1b674b8aa417\" (UID: \"71e3a1f4-081a-47c1-9cbe-1b674b8aa417\") " Oct 31 00:48:01.256149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.248778 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v" pod "71e3a1f4-081a-47c1-9cbe-1b674b8aa417" (UID: "71e3a1f4-081a-47c1-9cbe-1b674b8aa417"). InnerVolumeSpecName "kube-api-access-sgm9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:01.285467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.285425 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:48:01.291566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.291538 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:48:01.338691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.338659 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgm9v\" (UniqueName: \"kubernetes.io/projected/71e3a1f4-081a-47c1-9cbe-1b674b8aa417-kube-api-access-sgm9v\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:48:01.599938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.599895 5011 generic.go:356] "Generic (PLEG): container finished" podID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" containerID="efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa" exitCode=137 Oct 31 00:48:01.600537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.600512 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:48:01.600853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.600835 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:48:01.609606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.609583 5011 scope.go:122] "RemoveContainer" containerID="efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa" Oct 31 00:48:01.614512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.614176 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:48:01.635502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.632786 5011 scope.go:122] "RemoveContainer" containerID="efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa" Oct 31 00:48:01.635502 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:48:01.633183 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa\": container with ID starting with efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa not found: ID does not exist" containerID="efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa" Oct 31 00:48:01.635502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.633278 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa"} err="failed to get container status \"efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa\": rpc error: code = NotFound desc = could not find container \"efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa\": container with ID starting with efd80ef620da9fc1b60ed5cb5d28ddc5849f5eef60748eca8eeb032dcad5a2aa not found: ID does not exist" Oct 31 00:48:01.690230 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:48:01.686484 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-f1f0f8\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:48:01.757162 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:48:01.757124 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10893-f1f0f8\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" pod="opendatahub/test-curl-10893-f1f0f8" Oct 31 00:48:01.770198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:01.770171 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e3a1f4-081a-47c1-9cbe-1b674b8aa417" path="/var/lib/kubelet/pods/71e3a1f4-081a-47c1-9cbe-1b674b8aa417/volumes" Oct 31 00:48:01.952827 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:48:01.952726 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:48:04.071290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.070481 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.104229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.104135 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.128232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.106008 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.128232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.125415 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.168231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.167067 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.168231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.167460 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:48:04.168231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.168050 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.183559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.183294 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.401289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.398301 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.592238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.591297 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.592238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.591691 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:48:04.633230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.632323 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 00:48:04.708282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.703303 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.890754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:04.890714 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 00:48:05.081867 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:05.080276 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:05.081867 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:05.080673 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:48:05.113700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:05.113383 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:48:11.054021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:11.053982 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh"] Oct 31 00:48:11.120225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:11.108323 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a56df4-j9bfh"] Oct 31 00:48:11.688674 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:11.688638 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62b60de-2fa3-4352-bde4-2940889ef475" path="/var/lib/kubelet/pods/b62b60de-2fa3-4352-bde4-2940889ef475/volumes" Oct 31 00:48:11.956444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:11.956352 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:11.956444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:11.956418 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:13.126874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.126837 5011 generic.go:356] "Generic (PLEG): container finished" podID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerID="43072c510fb38961ff94e7589fb2210cb2a217a74074c7b95fda947da511a5e3" exitCode=0 Oct 31 00:48:13.127335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.126925 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerDied","Data":"43072c510fb38961ff94e7589fb2210cb2a217a74074c7b95fda947da511a5e3"} Oct 31 00:48:13.127465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.127446 5011 scope.go:122] "RemoveContainer" containerID="43072c510fb38961ff94e7589fb2210cb2a217a74074c7b95fda947da511a5e3" Oct 31 00:48:13.129684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.129644 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:13.129795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.129738 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:13.129795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.129783 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:13.368923 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.362669 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:48:13.373253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.373165 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.395014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.392265 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fa2c15c1-openshift-default-dockercfg-28j2p\"" Oct 31 00:48:13.410402 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.409770 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523290 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523364 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523408 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523435 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2r5p\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523587 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523645 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523743 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523786 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.525230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.523835 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652286 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652360 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652455 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652498 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652544 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652758 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652795 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.652823 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2r5p\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.653886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.653664 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.654750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.654144 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.654750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.654395 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.657759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.657733 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.699269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.673088 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.699269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.676242 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.716787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.716655 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2r5p\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.731230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.720881 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:13.732146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:13.732117 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\") pod \"e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:14.031283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.031038 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:14.218133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.217801 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerStarted","Data":"b60361993b37160bd36c64ff1491a885a0f1d4f4bc2004b0e6c2a953aecced0f"} Oct 31 00:48:14.406387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.406355 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:48:14.419585 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:48:14.419553 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657f029d_a7b4_4fc7_844b_6a74bf714ce5.slice/crio-7596bfd4e8553e6aa50ba52acc102f216f2099ffe4acb6d66c5c8d2e5415204f WatchSource:0}: Error finding container 7596bfd4e8553e6aa50ba52acc102f216f2099ffe4acb6d66c5c8d2e5415204f: Status 404 returned error can't find the container with id 7596bfd4e8553e6aa50ba52acc102f216f2099ffe4acb6d66c5c8d2e5415204f Oct 31 00:48:14.423098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.423062 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:14.423263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.423133 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:14.423263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.423163 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:14.947229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.946647 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:48:14.947229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.946779 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:14.947229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:14.946825 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:15.064297 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.064256 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd"] Oct 31 00:48:15.076301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.076201 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47038305-8cxqd"] Oct 31 00:48:15.249351 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.249255 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerStarted","Data":"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79"} Oct 31 00:48:15.249897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.249877 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerStarted","Data":"7596bfd4e8553e6aa50ba52acc102f216f2099ffe4acb6d66c5c8d2e5415204f"} Oct 31 00:48:15.300706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.300241 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podStartSLOduration=2.300195485 podStartE2EDuration="2.300195485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:15.294854665 +0000 UTC m=+1763.743942305" watchObservedRunningTime="2026-10-31 00:48:15.300195485 +0000 UTC m=+1763.749283123" Oct 31 00:48:15.688949 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.688915 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0daf7b5-680e-4dd4-b240-eee98313c19b" path="/var/lib/kubelet/pods/f0daf7b5-680e-4dd4-b240-eee98313c19b/volumes" Oct 31 00:48:15.956285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.955587 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:15.956285 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:15.955652 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:16.036549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.036515 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:16.055265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.051458 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:16.313708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.313631 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:16.314746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.314721 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:48:17.000233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.985352 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:17.000233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:16.985418 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:17.952228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:17.951798 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:17.952228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:17.951859 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:18.946643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:18.946604 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:18.946929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:18.946898 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:19.071856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:19.071795 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm"] Oct 31 00:48:19.074784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:19.074752 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f5a410a-mhgdm"] Oct 31 00:48:19.689662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:19.689300 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b50d89-eb62-46c0-8627-0cc9d7f001e4" path="/var/lib/kubelet/pods/43b50d89-eb62-46c0-8627-0cc9d7f001e4/volumes" Oct 31 00:48:19.958286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:19.957444 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:48:19.958286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:19.957515 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 00:48:20.950926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:20.950895 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:48:20.951520 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:20.951451 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:48:20.952251 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:20.952190 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 00:48:23.032910 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:23.032879 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn"] Oct 31 00:48:23.042876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:23.042847 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad8df849-n9knn"] Oct 31 00:48:23.693276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:23.693240 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907c6864-8500-4b31-9e5d-e6010e824b78" path="/var/lib/kubelet/pods/907c6864-8500-4b31-9e5d-e6010e824b78/volumes" Oct 31 00:48:25.036052 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:25.036014 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d"] Oct 31 00:48:25.046019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:25.045993 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87ee625-9lp8d"] Oct 31 00:48:25.686018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:25.685981 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630e2837-390c-462b-9d0a-466237913999" path="/var/lib/kubelet/pods/630e2837-390c-462b-9d0a-466237913999/volumes" Oct 31 00:48:30.031628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.031592 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww"] Oct 31 00:48:30.043425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.043396 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc0ea72-dq5ww"] Oct 31 00:48:30.453314 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.453279 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:48:30.459644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.459618 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.479105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.479081 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-14e039-serving-cert\"" Oct 31 00:48:30.479310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.479137 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:48:30.584082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.584047 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.584292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.584267 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6wh\" (UniqueName: \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.584461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.584442 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.584561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.584547 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.686037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.685999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.686194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.686106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.686194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.686178 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.686331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.686314 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6wh\" (UniqueName: \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.686749 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.686718 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.688760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.688734 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.688891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.688845 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.738283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.722530 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6wh\" (UniqueName: \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\") pod \"maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:30.778780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:30.778732 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:31.111148 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:48:31.111114 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a79c53d_2463_4aef_8d4d_f376f978ab80.slice/crio-ef95e39d2dc941f3f1bf0ab152b517002b73a7a34e2a6170c379b3f3cae4b018 WatchSource:0}: Error finding container ef95e39d2dc941f3f1bf0ab152b517002b73a7a34e2a6170c379b3f3cae4b018: Status 404 returned error can't find the container with id ef95e39d2dc941f3f1bf0ab152b517002b73a7a34e2a6170c379b3f3cae4b018 Oct 31 00:48:31.112568 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:31.112544 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:48:31.544905 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:31.544870 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" event={"ID":"8a79c53d-2463-4aef-8d4d-f376f978ab80","Type":"ContainerStarted","Data":"ef95e39d2dc941f3f1bf0ab152b517002b73a7a34e2a6170c379b3f3cae4b018"} Oct 31 00:48:31.695459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:31.695412 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e1b1fa-4d94-4332-9f3b-9b588365d119" path="/var/lib/kubelet/pods/13e1b1fa-4d94-4332-9f3b-9b588365d119/volumes" Oct 31 00:48:31.949371 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:48:31.949330 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:48:32.559122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:32.559076 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" event={"ID":"8a79c53d-2463-4aef-8d4d-f376f978ab80","Type":"ContainerStarted","Data":"e4c76d89b0c5fc87eadc79159541028d3dc98e21b3ece6335978feb87dd85a42"} Oct 31 00:48:32.559615 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:32.559168 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:32.627071 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:32.627008 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" podStartSLOduration=2.256270315 podStartE2EDuration="2.626988913s" totalImagesPullingTime="370.718598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:32.614961789 +0000 UTC m=+1781.064049427" watchObservedRunningTime="2026-10-31 00:48:32.626988913 +0000 UTC m=+1781.076076552" Oct 31 00:48:38.574966 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:38.574935 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:48:41.048630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:41.048589 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc"] Oct 31 00:48:41.059930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:41.059901 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b315f313-dwqxc"] Oct 31 00:48:41.713886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:41.713834 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd2117d-23db-4a81-962b-27e138695eaa" path="/var/lib/kubelet/pods/9fd2117d-23db-4a81-962b-27e138695eaa/volumes" Oct 31 00:48:46.052370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:46.052327 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk"] Oct 31 00:48:46.060037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:46.059607 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8773200-bj8zk"] Oct 31 00:48:47.031385 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:47.031353 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47"] Oct 31 00:48:47.043699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:47.043664 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e107f-96x47"] Oct 31 00:48:47.685946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:47.685911 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a030669-82b1-4d51-bb50-dc869dfe19d0" path="/var/lib/kubelet/pods/2a030669-82b1-4d51-bb50-dc869dfe19d0/volumes" Oct 31 00:48:47.686649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:47.686629 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c808da-0d3e-4797-ab28-a19788b42fa9" path="/var/lib/kubelet/pods/69c808da-0d3e-4797-ab28-a19788b42fa9/volumes" Oct 31 00:48:51.041608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:51.041563 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj"] Oct 31 00:48:51.053900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:51.053866 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be39919-2znqj"] Oct 31 00:48:51.685618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:51.685588 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62198c3a-093b-45e4-8b86-324b35e6c5ff" path="/var/lib/kubelet/pods/62198c3a-093b-45e4-8b86-324b35e6c5ff/volumes" Oct 31 00:48:52.011114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.011024 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:48:52.019872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.019839 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.041037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.041004 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:48:52.048502 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.048476 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-bcdb4e-openshift-default-dockercfg-s86d8\"" Oct 31 00:48:52.103308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103268 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103325 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103402 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103455 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103501 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103672 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103810 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103874 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103860 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.103929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.103902 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd2f\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.145321 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.145293 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:48:52.145820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.145798 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:48:52.205296 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205259 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd2f\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205448 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205420 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205471 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205555 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205595 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205833 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205716 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205838 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.205888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205852 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.206022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.205986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.206022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.206008 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.206363 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.206334 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.207907 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.207880 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.208170 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.208138 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.228357 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.228334 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd2f\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.228529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.228508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.307960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.307885 5011 scope.go:122] "RemoveContainer" containerID="1b0dd36d94d6b7e4322f5d1d6f3e1289468f8319c1a7c8e72406b3b686abb69a" Oct 31 00:48:52.337354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.337316 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:52.681857 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:48:52.681825 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8652ffe7_53fd_4b05_a202_9b51be480c88.slice/crio-7d8b8bedcd68091048d1ca3ea56ea8bf8d58572b0b2f185ff777b0e8ae2c8192 WatchSource:0}: Error finding container 7d8b8bedcd68091048d1ca3ea56ea8bf8d58572b0b2f185ff777b0e8ae2c8192: Status 404 returned error can't find the container with id 7d8b8bedcd68091048d1ca3ea56ea8bf8d58572b0b2f185ff777b0e8ae2c8192 Oct 31 00:48:52.682158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.682136 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:48:52.685313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.685277 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:52.685395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.685360 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:52.685434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:52.685402 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:48:53.191229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.183717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" event={"ID":"8652ffe7-53fd-4b05-a202-9b51be480c88","Type":"ContainerStarted","Data":"99055bd1bccf219af7b18a7292f08298b227385fb0ad29c277d4a5340a601fa4"} Oct 31 00:48:53.191229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.183782 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" event={"ID":"8652ffe7-53fd-4b05-a202-9b51be480c88","Type":"ContainerStarted","Data":"7d8b8bedcd68091048d1ca3ea56ea8bf8d58572b0b2f185ff777b0e8ae2c8192"} Oct 31 00:48:53.309191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.309115 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podStartSLOduration=2.309089438 podStartE2EDuration="2.309089438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:53.276731856 +0000 UTC m=+1801.725819496" watchObservedRunningTime="2026-10-31 00:48:53.309089438 +0000 UTC m=+1801.758177078" Oct 31 00:48:53.338846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.338814 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:53.345247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.341452 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:53.345247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:53.341509 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:54.349233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:54.340315 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:54.349233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:54.340397 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:55.342232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:55.339542 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.342232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:55.339616 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:56.348231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:56.340607 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:56.348231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:56.340690 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:57.341559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:57.341520 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:57.341833 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:57.341806 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:58.054600 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:58.054556 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg"] Oct 31 00:48:58.058885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:58.058859 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4b9d96-k2wwg"] Oct 31 00:48:58.345230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:58.339595 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:48:58.345230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:58.339674 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 00:48:59.342842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:59.342813 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:59.343353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:59.343321 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:59.344086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:59.344067 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:48:59.684991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:48:59.684912 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d" path="/var/lib/kubelet/pods/feff3e32-1c27-4d0f-a9a1-7ec5fba1ca6d/volumes" Oct 31 00:49:01.949189 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:01.949150 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:49:02.035919 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:02.035876 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8"] Oct 31 00:49:02.045382 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:02.045355 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5813c6c-mw6r8"] Oct 31 00:49:03.686083 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:03.686046 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a556e5d-f3a8-43a4-b325-96a116ef5555" path="/var/lib/kubelet/pods/2a556e5d-f3a8-43a4-b325-96a116ef5555/volumes" Oct 31 00:49:06.658121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:06.658084 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:49:15.035325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:15.035289 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6"] Oct 31 00:49:15.059724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:15.059690 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e236238-7h4d6"] Oct 31 00:49:15.685945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:15.685753 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6baacca0-71e8-4389-a33e-a05602e81395" path="/var/lib/kubelet/pods/6baacca0-71e8-4389-a33e-a05602e81395/volumes" Oct 31 00:49:17.358051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.356944 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:17.358051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.357013 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.642444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.642354 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:17.642444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.642432 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.774368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.774333 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10896-0bcee6"] Oct 31 00:49:17.809896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.809863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-0bcee6"] Oct 31 00:49:17.810128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.809983 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:17.826094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.826048 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\"" Oct 31 00:49:17.947862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:17.947782 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46mb\" (UniqueName: \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\") pod \"test-curl-10896-0bcee6\" (UID: \"4f15b452-2002-42be-ba0c-21294359794f\") " pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:18.050142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.050096 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46mb\" (UniqueName: \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\") pod \"test-curl-10896-0bcee6\" (UID: \"4f15b452-2002-42be-ba0c-21294359794f\") " pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:18.075490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.075455 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46mb\" (UniqueName: \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\") pod \"test-curl-10896-0bcee6\" (UID: \"4f15b452-2002-42be-ba0c-21294359794f\") " pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:18.152748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.152703 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:18.504768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.504729 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-0bcee6"] Oct 31 00:49:18.505312 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:49:18.505283 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f15b452_2002_42be_ba0c_21294359794f.slice/crio-2eda8f4696d7452db93ba83754b24ca26ba0bd78a578979e13ada034fa61ee0b WatchSource:0}: Error finding container 2eda8f4696d7452db93ba83754b24ca26ba0bd78a578979e13ada034fa61ee0b: Status 404 returned error can't find the container with id 2eda8f4696d7452db93ba83754b24ca26ba0bd78a578979e13ada034fa61ee0b Oct 31 00:49:18.708760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.708715 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-0bcee6" event={"ID":"4f15b452-2002-42be-ba0c-21294359794f","Type":"ContainerStarted","Data":"e1dbe11d28404531d6df76553a79c276c369a605b6bdbfd2b3975b661a520ddb"} Oct 31 00:49:18.708760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.708765 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-0bcee6" event={"ID":"4f15b452-2002-42be-ba0c-21294359794f","Type":"ContainerStarted","Data":"2eda8f4696d7452db93ba83754b24ca26ba0bd78a578979e13ada034fa61ee0b"} Oct 31 00:49:18.749154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:18.749093 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10896-0bcee6" podStartSLOduration=1.749074014 podStartE2EDuration="1.749074014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:18.747201567 +0000 UTC m=+1827.196289212" watchObservedRunningTime="2026-10-31 00:49:18.749074014 +0000 UTC m=+1827.198161653" Oct 31 00:49:19.818231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:19.808985 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10896-0bcee6"] Oct 31 00:49:19.818231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:19.816488 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10896-0bcee6"] Oct 31 00:49:20.065062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.063672 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7"] Oct 31 00:49:20.073646 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.073579 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61925b28-dwlg7"] Oct 31 00:49:20.527417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.527323 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f"] Oct 31 00:49:20.528464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.528440 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4f15b452-2002-42be-ba0c-21294359794f" containerName="test-curl-10896-0bcee6" Oct 31 00:49:20.528464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.528464 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f15b452-2002-42be-ba0c-21294359794f" containerName="test-curl-10896-0bcee6" Oct 31 00:49:20.528641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.528628 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f15b452-2002-42be-ba0c-21294359794f" containerName="test-curl-10896-0bcee6" Oct 31 00:49:20.532902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.532881 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.558254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.552079 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f"] Oct 31 00:49:20.563513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.563475 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:20.564173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.564152 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:49:20.713553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.713517 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvjf\" (UniqueName: \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.713795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.713773 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.713869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.713827 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.746451 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.746419 5011 generic.go:356] "Generic (PLEG): container finished" podID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerID="5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79" exitCode=0 Oct 31 00:49:20.746581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.746488 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerDied","Data":"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79"} Oct 31 00:49:20.747013 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.746990 5011 scope.go:122] "RemoveContainer" containerID="5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79" Oct 31 00:49:20.749266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749235 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.749266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749256 5011 generic.go:356] "Generic (PLEG): container finished" podID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerID="d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7" exitCode=0 Oct 31 00:49:20.749436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749324 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.749436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749381 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.749436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749326 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerDied","Data":"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7"} Oct 31 00:49:20.749588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749507 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10896-0bcee6" podUID="4f15b452-2002-42be-ba0c-21294359794f" containerName="test-curl-10896-0bcee6" containerID="cri-o://e1dbe11d28404531d6df76553a79c276c369a605b6bdbfd2b3975b661a520ddb" gracePeriod=2 Oct 31 00:49:20.749824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.749759 5011 scope.go:122] "RemoveContainer" containerID="d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7" Oct 31 00:49:20.751732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.751702 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.751820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.751802 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.751866 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.751848 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:20.809614 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:20.809574 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-0bcee6\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="4f15b452-2002-42be-ba0c-21294359794f" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:20.819920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.817352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.819920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.819084 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.819920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.819349 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvjf\" (UniqueName: \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.819920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.819876 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.827642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.825596 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.844357 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.844279 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvjf\" (UniqueName: \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:20.869853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:20.869824 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:21.271147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:21.271116 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f"] Oct 31 00:49:21.284327 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:49:21.277868 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e1907e_5b52_4a6a_8418_a47327d24f69.slice/crio-78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f WatchSource:0}: Error finding container 78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f: Status 404 returned error can't find the container with id 78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f Oct 31 00:49:21.687511 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:21.687473 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efe2d40-b8bf-4089-8c4c-4671a008eaf7" path="/var/lib/kubelet/pods/4efe2d40-b8bf-4089-8c4c-4671a008eaf7/volumes" Oct 31 00:49:21.695259 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:21.695219 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-0bcee6\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="4f15b452-2002-42be-ba0c-21294359794f" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:21.771253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:21.771182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" event={"ID":"d7e1907e-5b52-4a6a-8418-a47327d24f69","Type":"ContainerStarted","Data":"78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f"} Oct 31 00:49:21.773259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:21.773231 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerStarted","Data":"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e"} Oct 31 00:49:21.775372 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:21.775343 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerStarted","Data":"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028"} Oct 31 00:49:22.179930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.179883 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:49:22.219258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.201800 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:22.219258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.215319 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:22.665226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.651795 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:22.665226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.651872 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:22.742228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.737588 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:49:22.778280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.771241 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:49:22.778280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.772022 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.796351 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.794160 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:49:22.813097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.813068 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3d21c7-openshift-default-dockercfg-zt7qx\"" Oct 31 00:49:22.824941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.824890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.825257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.825238 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.827417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.827395 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.827809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.827789 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.828012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.827994 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnnc\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.828222 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.828191 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.828444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.828427 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.828706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.828689 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.828896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.828881 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.835496 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.835471 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" event={"ID":"d7e1907e-5b52-4a6a-8418-a47327d24f69","Type":"ContainerStarted","Data":"6b7ff9875c1c02fc151f77ed967a668e5e91b26a5800ee1e27fdfff6523a99cc"} Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934489 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934532 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934701 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934726 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.934797 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.935051 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.935106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnnc\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.939002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.935192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.952244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.945964 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.952244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.948706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.965278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.963446 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.965278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.964048 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.977281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.967252 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:22.984302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.982913 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" podStartSLOduration=2.982888915 podStartE2EDuration="2.982888915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:22.931071291 +0000 UTC m=+1831.380158931" watchObservedRunningTime="2026-10-31 00:49:22.982888915 +0000 UTC m=+1831.431976556" Oct 31 00:49:22.992230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:22.989488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:23.025557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.025523 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:23.041545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.040862 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:23.041545 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.040926 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:23.042964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.042901 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnnc\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:23.043401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.043346 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\") pod \"e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:23.137263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.137231 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:23.286588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.286007 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:23.286588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.286085 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:23.879334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.879294 5011 generic.go:356] "Generic (PLEG): container finished" podID="4f15b452-2002-42be-ba0c-21294359794f" containerID="e1dbe11d28404531d6df76553a79c276c369a605b6bdbfd2b3975b661a520ddb" exitCode=137 Oct 31 00:49:23.882443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.882415 5011 generic.go:356] "Generic (PLEG): container finished" podID="d7e1907e-5b52-4a6a-8418-a47327d24f69" containerID="6b7ff9875c1c02fc151f77ed967a668e5e91b26a5800ee1e27fdfff6523a99cc" exitCode=0 Oct 31 00:49:23.882632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:23.882613 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" event={"ID":"d7e1907e-5b52-4a6a-8418-a47327d24f69","Type":"ContainerDied","Data":"6b7ff9875c1c02fc151f77ed967a668e5e91b26a5800ee1e27fdfff6523a99cc"} Oct 31 00:49:24.087621 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.085778 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:24.087621 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.085845 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:24.122379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.122335 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:24.123851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.123816 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:24.178345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.178313 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:24.320519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.294096 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\" (UniqueName: \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\") pod \"4f15b452-2002-42be-ba0c-21294359794f\" (UID: \"4f15b452-2002-42be-ba0c-21294359794f\") " Oct 31 00:49:24.386588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.386538 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:24.421916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.386844 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:24.500695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.500623 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb" pod "4f15b452-2002-42be-ba0c-21294359794f" (UID: "4f15b452-2002-42be-ba0c-21294359794f"). InnerVolumeSpecName "kube-api-access-r46mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:24.549320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.549274 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4"] Oct 31 00:49:24.585472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.569711 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46mb\" (UniqueName: \"kubernetes.io/projected/4f15b452-2002-42be-ba0c-21294359794f-kube-api-access-r46mb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:24.664293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.650403 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c04e1f-zgqm4"] Oct 31 00:49:24.665254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.665189 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:24.665519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.665496 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:24.665713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.665694 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:49:24.700675 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.700637 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:49:24.931603 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.931506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerStarted","Data":"7f2569fa96f394d3b853aaa74c3c33633590f83208a5d1f69773c13c7b63dd95"} Oct 31 00:49:24.991252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:24.980834 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:25.005234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.001578 5011 scope.go:122] "RemoveContainer" containerID="e1dbe11d28404531d6df76553a79c276c369a605b6bdbfd2b3975b661a520ddb" Oct 31 00:49:25.043169 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.043139 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:25.043430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.043386 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:25.079053 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:25.078995 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-0bcee6\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="4f15b452-2002-42be-ba0c-21294359794f" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:25.136121 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.136073 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:49:25.136998 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.136969 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:49:25.158635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.158319 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.175387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.175351 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.178142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.177414 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.185412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.184289 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:49:25.220697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.218301 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.267247 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:25.249521 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-0bcee6\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="4f15b452-2002-42be-ba0c-21294359794f" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:25.368727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.368689 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:25.401129 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.401078 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:25.441283 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:25.434426 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-0bcee6\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="4f15b452-2002-42be-ba0c-21294359794f" pod="opendatahub/test-curl-10896-0bcee6" Oct 31 00:49:25.648370 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.648321 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.804951 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.803383 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:25.945231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.916190 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb4d46d-477f-4bc3-975d-edefb7cb6249" path="/var/lib/kubelet/pods/1eb4d46d-477f-4bc3-975d-edefb7cb6249/volumes" Oct 31 00:49:25.945231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.916871 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f15b452-2002-42be-ba0c-21294359794f" path="/var/lib/kubelet/pods/4f15b452-2002-42be-ba0c-21294359794f/volumes" Oct 31 00:49:25.945231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:25.938701 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:26.100712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.100676 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:26.100992 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.100962 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:26.114274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.107723 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 00:49:26.114274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.108722 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 00:49:26.394523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.388491 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:26.394523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.392679 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:49:26.405419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.405370 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:26.405655 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.405445 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:26.516564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.516527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerStarted","Data":"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4"} Oct 31 00:49:26.642091 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.614711 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:49:26.966659 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.966621 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:27.000233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:26.998756 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podStartSLOduration=5.998733061 podStartE2EDuration="5.998733061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:26.735117899 +0000 UTC m=+1835.184205538" watchObservedRunningTime="2026-10-31 00:49:26.998733061 +0000 UTC m=+1835.447820703" Oct 31 00:49:27.007460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.007410 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\" (UniqueName: \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\") pod \"d7e1907e-5b52-4a6a-8418-a47327d24f69\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " Oct 31 00:49:27.007955 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.007908 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\") pod \"d7e1907e-5b52-4a6a-8418-a47327d24f69\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " Oct 31 00:49:27.008260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.008236 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\") pod \"d7e1907e-5b52-4a6a-8418-a47327d24f69\" (UID: \"d7e1907e-5b52-4a6a-8418-a47327d24f69\") " Oct 31 00:49:27.027230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.013387 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca" pod "d7e1907e-5b52-4a6a-8418-a47327d24f69" (UID: "d7e1907e-5b52-4a6a-8418-a47327d24f69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:27.080225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.069116 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca" pod "d7e1907e-5b52-4a6a-8418-a47327d24f69" (UID: "d7e1907e-5b52-4a6a-8418-a47327d24f69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:27.080225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.072570 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:27.080225 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.072640 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:27.093758 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.093721 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf" pod "d7e1907e-5b52-4a6a-8418-a47327d24f69" (UID: "d7e1907e-5b52-4a6a-8418-a47327d24f69"). InnerVolumeSpecName "kube-api-access-fmvjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:27.119642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.112918 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:27.119642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.112951 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvjf\" (UniqueName: \"kubernetes.io/projected/d7e1907e-5b52-4a6a-8418-a47327d24f69-kube-api-access-fmvjf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:27.119642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.112966 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1907e-5b52-4a6a-8418-a47327d24f69-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:27.155531 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.155275 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:27.197015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.196978 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:27.289274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.285476 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:27.289274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.285533 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:27.687228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.674327 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" event={"ID":"d7e1907e-5b52-4a6a-8418-a47327d24f69","Type":"ContainerDied","Data":"78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f"} Oct 31 00:49:27.687228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.674412 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e9ae416a768df65b85b7bb6750654546c6e6cc9effa4284c315ce30235e55f" Oct 31 00:49:27.687228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.674505 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f" Oct 31 00:49:27.744589 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.742083 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:27.744589 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:27.742181 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:49:28.040723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:28.040636 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:28.040983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:28.040951 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:28.285753 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:28.285711 5011 patch_prober.go:29] interesting pod/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:49:28.285928 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:28.285766 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 00:49:29.037616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.037535 5011 patch_prober.go:29] interesting pod/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:49:29.037616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.037598 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 00:49:29.054503 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.054454 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7"] Oct 31 00:49:29.061719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.061678 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874ab0e4-chdz7"] Oct 31 00:49:29.321365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.316581 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:49:29.321365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.318079 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:49:29.321365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.320150 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:49:29.686100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:29.686016 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fde329c-96d4-4e48-bc77-97125cfa5ae2" path="/var/lib/kubelet/pods/5fde329c-96d4-4e48-bc77-97125cfa5ae2/volumes" Oct 31 00:49:30.037487 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.037410 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:49:30.037812 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.037787 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:49:30.038719 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.038695 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:49:30.309668 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.309595 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw"] Oct 31 00:49:30.310384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.310367 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d7e1907e-5b52-4a6a-8418-a47327d24f69" containerName="revoke-keys" Oct 31 00:49:30.310460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.310384 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e1907e-5b52-4a6a-8418-a47327d24f69" containerName="revoke-keys" Oct 31 00:49:30.310509 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.310499 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e1907e-5b52-4a6a-8418-a47327d24f69" containerName="revoke-keys" Oct 31 00:49:30.315535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.315516 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.331886 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.331863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw"] Oct 31 00:49:30.345709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.345678 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvkl\" (UniqueName: \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.345849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.345811 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.345961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.345939 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.368839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.368810 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:30.368977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.368921 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:49:30.447824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.447782 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvkl\" (UniqueName: \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.447990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.447865 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.447990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.447932 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.448695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.448668 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.450413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.450392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.491476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.491443 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvkl\" (UniqueName: \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\") pod \"maas-api-revoke-sub-4dbb2511-f4dbw\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.633986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.633955 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:30.949754 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:49:30.949721 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe38a8e_0401_45cd_85d6_14cfce83ae1f.slice/crio-0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a WatchSource:0}: Error finding container 0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a: Status 404 returned error can't find the container with id 0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a Oct 31 00:49:30.950223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:30.950184 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw"] Oct 31 00:49:31.888380 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:31.888292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" event={"ID":"2fe38a8e-0401-45cd-85d6-14cfce83ae1f","Type":"ContainerStarted","Data":"fcc26afcab26044f1bfc9d4b4f8a27db65280882c4af8f9242c2e4e469349c44"} Oct 31 00:49:31.888380 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:31.888348 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" event={"ID":"2fe38a8e-0401-45cd-85d6-14cfce83ae1f","Type":"ContainerStarted","Data":"0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a"} Oct 31 00:49:31.957849 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:31.957795 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" podStartSLOduration=1.957780635 podStartE2EDuration="1.957780635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:31.939476799 +0000 UTC m=+1840.388564438" watchObservedRunningTime="2026-10-31 00:49:31.957780635 +0000 UTC m=+1840.406868274" Oct 31 00:49:31.959719 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:31.959685 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:49:33.024142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:33.023175 5011 generic.go:356] "Generic (PLEG): container finished" podID="2fe38a8e-0401-45cd-85d6-14cfce83ae1f" containerID="fcc26afcab26044f1bfc9d4b4f8a27db65280882c4af8f9242c2e4e469349c44" exitCode=0 Oct 31 00:49:33.024142 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:33.023534 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" event={"ID":"2fe38a8e-0401-45cd-85d6-14cfce83ae1f","Type":"ContainerDied","Data":"fcc26afcab26044f1bfc9d4b4f8a27db65280882c4af8f9242c2e4e469349c44"} Oct 31 00:49:34.243480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.243451 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:34.380677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.380640 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\") pod \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " Oct 31 00:49:34.380877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.380725 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\" (UniqueName: \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\") pod \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " Oct 31 00:49:34.380877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.380767 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\") pod \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\" (UID: \"2fe38a8e-0401-45cd-85d6-14cfce83ae1f\") " Oct 31 00:49:34.381159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.381130 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca" pod "2fe38a8e-0401-45cd-85d6-14cfce83ae1f" (UID: "2fe38a8e-0401-45cd-85d6-14cfce83ae1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:34.381592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.381570 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:34.382795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.382777 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca" pod "2fe38a8e-0401-45cd-85d6-14cfce83ae1f" (UID: "2fe38a8e-0401-45cd-85d6-14cfce83ae1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:34.382876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.382857 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl" pod "2fe38a8e-0401-45cd-85d6-14cfce83ae1f" (UID: "2fe38a8e-0401-45cd-85d6-14cfce83ae1f"). InnerVolumeSpecName "kube-api-access-frvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:34.483869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.483831 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:34.483869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:34.483870 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvkl\" (UniqueName: \"kubernetes.io/projected/2fe38a8e-0401-45cd-85d6-14cfce83ae1f-kube-api-access-frvkl\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:35.032267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.032226 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl"] Oct 31 00:49:35.042401 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.042363 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" event={"ID":"2fe38a8e-0401-45cd-85d6-14cfce83ae1f","Type":"ContainerDied","Data":"0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a"} Oct 31 00:49:35.042566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.042411 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcf48d710c565113cb645dea5ee0c6620ed1fb8d47339fc0e5cb51a05cf247a" Oct 31 00:49:35.042566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.042379 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw" Oct 31 00:49:35.045267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.045243 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5792bd42-llqbl"] Oct 31 00:49:35.685619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:35.685585 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6965bd9b-801e-43f0-8084-952e13e2df39" path="/var/lib/kubelet/pods/6965bd9b-801e-43f0-8084-952e13e2df39/volumes" Oct 31 00:49:36.227272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.227240 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5"] Oct 31 00:49:36.227916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.227902 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2fe38a8e-0401-45cd-85d6-14cfce83ae1f" containerName="revoke-keys" Oct 31 00:49:36.227964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.227917 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe38a8e-0401-45cd-85d6-14cfce83ae1f" containerName="revoke-keys" Oct 31 00:49:36.228049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.228040 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe38a8e-0401-45cd-85d6-14cfce83ae1f" containerName="revoke-keys" Oct 31 00:49:36.232056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.232029 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.250641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.250608 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5"] Oct 31 00:49:36.255693 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.255670 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:36.255858 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.255746 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:49:36.415526 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.415489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.415808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.415787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.415945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.415928 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.517067 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.516989 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.517231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.517116 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.517231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.517168 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.517853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.517826 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.519617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.519592 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.540919 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.540893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\") pod \"maas-api-revoke-sub-461c4443-vrfp5\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.560618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.560592 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:36.921594 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:49:36.921562 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10398cf9_a1f3_470b_8dd6_cab5c17e7434.slice/crio-207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd WatchSource:0}: Error finding container 207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd: Status 404 returned error can't find the container with id 207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd Oct 31 00:49:36.925027 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:36.924995 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5"] Oct 31 00:49:37.061970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:37.061933 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" event={"ID":"10398cf9-a1f3-470b-8dd6-cab5c17e7434","Type":"ContainerStarted","Data":"207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd"} Oct 31 00:49:38.071940 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:38.071899 5011 generic.go:356] "Generic (PLEG): container finished" podID="10398cf9-a1f3-470b-8dd6-cab5c17e7434" containerID="d78f5d24b6436bd2ac63cd1c9fd392e233e49db363c1aeece9d448e123982762" exitCode=0 Oct 31 00:49:38.072368 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:38.071953 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" event={"ID":"10398cf9-a1f3-470b-8dd6-cab5c17e7434","Type":"ContainerDied","Data":"d78f5d24b6436bd2ac63cd1c9fd392e233e49db363c1aeece9d448e123982762"} Oct 31 00:49:39.314438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.314409 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:39.379605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.379525 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\") pod \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " Oct 31 00:49:39.379770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.379697 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\") pod \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " Oct 31 00:49:39.379897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.379874 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\") pod \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\" (UID: \"10398cf9-a1f3-470b-8dd6-cab5c17e7434\") " Oct 31 00:49:39.380555 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.380523 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca" pod "10398cf9-a1f3-470b-8dd6-cab5c17e7434" (UID: "10398cf9-a1f3-470b-8dd6-cab5c17e7434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:39.381246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.381221 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:39.398367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.398337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl" pod "10398cf9-a1f3-470b-8dd6-cab5c17e7434" (UID: "10398cf9-a1f3-470b-8dd6-cab5c17e7434"). InnerVolumeSpecName "kube-api-access-cbrtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:39.399514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.399492 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca" pod "10398cf9-a1f3-470b-8dd6-cab5c17e7434" (UID: "10398cf9-a1f3-470b-8dd6-cab5c17e7434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:39.483256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.483226 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10398cf9-a1f3-470b-8dd6-cab5c17e7434-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:39.483256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:39.483253 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/10398cf9-a1f3-470b-8dd6-cab5c17e7434-kube-api-access-cbrtl\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:49:40.091920 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:40.091887 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" Oct 31 00:49:40.092072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:40.091921 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5" event={"ID":"10398cf9-a1f3-470b-8dd6-cab5c17e7434","Type":"ContainerDied","Data":"207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd"} Oct 31 00:49:40.092072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:40.091971 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207b6ff6265d5a2773fdde5653f116accd894123cc5e7da4d0472344b337f9dd" Oct 31 00:49:48.037019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.036978 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:49:48.038653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.038625 5011 cpu_manager.go:414] "containerMap: removing container" podUID="10398cf9-a1f3-470b-8dd6-cab5c17e7434" containerName="revoke-keys" Oct 31 00:49:48.038653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.038652 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10398cf9-a1f3-470b-8dd6-cab5c17e7434" containerName="revoke-keys" Oct 31 00:49:48.039100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.039085 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="10398cf9-a1f3-470b-8dd6-cab5c17e7434" containerName="revoke-keys" Oct 31 00:49:48.051238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.051170 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.062782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.062747 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:49:48.068566 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.068532 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3b1945-serving-cert\"" Oct 31 00:49:48.147034 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.146993 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.147301 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.147279 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.147355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.147318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.147355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.147339 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82v8\" (UniqueName: \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.248557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.248519 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.248765 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.248739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.248835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.248769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.248835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.248789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82v8\" (UniqueName: \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.249087 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:48.249062 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-serving-cert: secret "maas-api-e2e-api-b-3b1945-serving-cert" not found Oct 31 00:49:48.249279 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:49:48.249266 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls podName:0b80780e-6fb8-4f48-bd07-e1c8f4f73c15 nodeName:}" failed. No retries permitted until 2026-10-31 00:49:48.749244012 +0000 UTC m=+1857.198331632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls") pod "maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" (UID: "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15") : secret "maas-api-e2e-api-b-3b1945-serving-cert" not found Oct 31 00:49:48.249560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.249539 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.253248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.253175 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.278439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.278411 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82v8\" (UniqueName: \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.757113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.757076 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.759902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.759850 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") pod \"maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:48.987488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:48.987450 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:49.412706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:49.412675 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:49:49.431233 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:49:49.426640 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b80780e_6fb8_4f48_bd07_e1c8f4f73c15.slice/crio-bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b WatchSource:0}: Error finding container bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b: Status 404 returned error can't find the container with id bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b Oct 31 00:49:50.411748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:50.411712 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" event={"ID":"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15","Type":"ContainerStarted","Data":"811573f81bbee21ef524d424d76a71516cda461c1c2a0860e85af3e87d818b30"} Oct 31 00:49:50.412012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:50.411774 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:49:50.412012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:50.411797 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" event={"ID":"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15","Type":"ContainerStarted","Data":"bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b"} Oct 31 00:49:50.458683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:50.458624 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" podStartSLOduration=1.99254986 podStartE2EDuration="2.458606414s" totalImagesPullingTime="466.056554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:50.454972488 +0000 UTC m=+1858.904060130" watchObservedRunningTime="2026-10-31 00:49:50.458606414 +0000 UTC m=+1858.907694053" Oct 31 00:49:52.380868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.380834 5011 scope.go:122] "RemoveContainer" containerID="93d6d7244ad3f3b6b112b6113a4f0bb4ecea2f1e67cfb7003c1de4dbdec84e5a" Oct 31 00:49:52.391997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.391969 5011 scope.go:122] "RemoveContainer" containerID="ce1be5d51d6e6c2401dc6919b3fbcd4bd57e4afd029d67144bd1a461d17cb827" Oct 31 00:49:52.401817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.401797 5011 scope.go:122] "RemoveContainer" containerID="95cc68dab97ede431881ce707dbacddb6588e391794a020f7f3b8efd35e5da0b" Oct 31 00:49:52.411371 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.411348 5011 scope.go:122] "RemoveContainer" containerID="975ac4d14bba38543bf9107acd6ba1645144beb9da0016656088ca36b53865af" Oct 31 00:49:52.422813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.422794 5011 scope.go:122] "RemoveContainer" containerID="d2e584d4ebe4f2e9196bb8c11d6148bb406bead176a846b91e48d9133ecea169" Oct 31 00:49:52.434110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.434092 5011 scope.go:122] "RemoveContainer" containerID="006e1a8ff4a33759d627e495f391266205b6bfdd230b571f2839364faa81a6e1" Oct 31 00:49:52.445641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.445610 5011 scope.go:122] "RemoveContainer" containerID="f01c2dfd9a5f36327f6511cbaf0f2574c81b0ef4b11c8dfa17bf339bd53dd6c8" Oct 31 00:49:52.456681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.456661 5011 scope.go:122] "RemoveContainer" containerID="c6ae5e7f69b0055ba0db3d85c1916d4a530246fe81fcda01e8726718dc6505dc" Oct 31 00:49:52.467271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.467246 5011 scope.go:122] "RemoveContainer" containerID="44988df288e1b4e8e5dcad017969fdf1637bc67e31bcc8a2937e018bd3c44b89" Oct 31 00:49:52.476984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.476962 5011 scope.go:122] "RemoveContainer" containerID="8246145c2768d6a5da7bcdcf440a2c21bce7557cd6c1bcb273fb6d246386f374" Oct 31 00:49:52.862229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:52.862161 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:49:56.427100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:49:56.426929 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:50:01.948742 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:50:01.948699 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:50:17.207726 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:17.207686 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:50:17.208184 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:17.207990 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" podUID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" containerName="maas-api" containerID="cri-o://7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5" gracePeriod=30 Oct 31 00:50:17.977242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:17.967497 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:50:18.197452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.196816 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\" (UniqueName: \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\") pod \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " Oct 31 00:50:18.197452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.197189 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\") pod \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " Oct 31 00:50:18.197452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.197274 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") pod \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " Oct 31 00:50:18.197452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.197315 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\") pod \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\" (UID: \"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60\") " Oct 31 00:50:18.202977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.202932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca" pod "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:18.204575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.204531 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x" pod "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60"). InnerVolumeSpecName "kube-api-access-7vn5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:18.206715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.206687 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs" pod "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:18.207484 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.207458 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls" pod "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" (UID: "b8d567ec-9a8c-458a-91b9-b7c61fb8ca60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:18.211151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.211126 5011 generic.go:356] "Generic (PLEG): container finished" podID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" containerID="7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5" exitCode=0 Oct 31 00:50:18.211552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.211327 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" Oct 31 00:50:18.211552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.211340 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" event={"ID":"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60","Type":"ContainerDied","Data":"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5"} Oct 31 00:50:18.211552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.211381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8" event={"ID":"b8d567ec-9a8c-458a-91b9-b7c61fb8ca60","Type":"ContainerDied","Data":"6bb42e7300db9f44533181b2130586433a6f1781e80d0d0131491652f8fe68f2"} Oct 31 00:50:18.211552 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.211407 5011 scope.go:122] "RemoveContainer" containerID="7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5" Oct 31 00:50:18.251439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.251417 5011 scope.go:122] "RemoveContainer" containerID="7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5" Oct 31 00:50:18.251760 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:50:18.251737 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5\": container with ID starting with 7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5 not found: ID does not exist" containerID="7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5" Oct 31 00:50:18.251839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.251769 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5"} err="failed to get container status \"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5\": rpc error: code = NotFound desc = could not find container \"7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5\": container with ID starting with 7537971452f5d725c67ca9418281b98f6e32baa62f8b4feefd5bb6e3d5fb71f5 not found: ID does not exist" Oct 31 00:50:18.261845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.261817 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f"] Oct 31 00:50:18.266236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.266215 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mrb5f"] Oct 31 00:50:18.287935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.287911 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:50:18.291960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.291941 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-208e9f32-5cbcdcf9bf-2lvx8"] Oct 31 00:50:18.299196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.299167 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:18.299196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.299193 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:18.299360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.299219 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:18.299360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:18.299234 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vn5x\" (UniqueName: \"kubernetes.io/projected/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60-kube-api-access-7vn5x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:19.242287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.242254 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:50:19.242796 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.242473 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" containerID="cri-o://6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519" gracePeriod=30 Oct 31 00:50:19.248995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.248967 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:50:19.249252 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.249190 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" containerID="cri-o://b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19" gracePeriod=30 Oct 31 00:50:19.687172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.687140 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" path="/var/lib/kubelet/pods/b8d567ec-9a8c-458a-91b9-b7c61fb8ca60/volumes" Oct 31 00:50:19.687689 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:19.687672 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e1907e-5b52-4a6a-8418-a47327d24f69" path="/var/lib/kubelet/pods/d7e1907e-5b52-4a6a-8418-a47327d24f69/volumes" Oct 31 00:50:24.511574 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.511510 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:50:24.591191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.591163 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:50:24.615085 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615052 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615110 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615146 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615198 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615240 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615278 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615428 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615467 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615548 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615505 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615548 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615530 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615631 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615610 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615729 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615819 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.615861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615846 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615877 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.615970 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615937 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\") pod \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\" (UID: \"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0\") " Oct 31 00:50:24.616080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.615982 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\") pod \"18ac50da-c78d-489a-9fa1-686293c3ac84\" (UID: \"18ac50da-c78d-489a-9fa1-686293c3ac84\") " Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.616920 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.617137 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.617482 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.617805 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.618003 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.619620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.618798 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.620067 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.619988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.621637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.621102 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.621637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.621274 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.621637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.621583 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.621637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.621596 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.625567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.625539 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:24.625710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.625684 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.625783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.625762 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "kube-api-access-lbnxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.625851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.625810 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.625919 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.625894 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "kube-api-access-v9j2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.627257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.627165 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo" pod "18ac50da-c78d-489a-9fa1-686293c3ac84" (UID: "18ac50da-c78d-489a-9fa1-686293c3ac84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:24.628477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.628454 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy" pod "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" (UID: "2fd667b8-103b-44e9-b8b7-9f633ff7c4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.718613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718575 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718606 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718618 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718626 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718635 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnxh\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-kube-api-access-lbnxh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718645 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718653 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718660 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ac50da-c78d-489a-9fa1-686293c3ac84-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718668 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9j2c\" (UniqueName: \"kubernetes.io/projected/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-kube-api-access-v9j2c\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718675 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718685 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718692 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718700 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718708 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ac50da-c78d-489a-9fa1-686293c3ac84-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718716 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718723 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718731 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.718859 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:24.718740 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ac50da-c78d-489a-9fa1-686293c3ac84-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.298644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.298605 5011 generic.go:356] "Generic (PLEG): container finished" podID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerID="6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519" exitCode=0 Oct 31 00:50:25.298778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.298656 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" Oct 31 00:50:25.298778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.298689 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" event={"ID":"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0","Type":"ContainerDied","Data":"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519"} Oct 31 00:50:25.298778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.298747 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7" event={"ID":"2fd667b8-103b-44e9-b8b7-9f633ff7c4f0","Type":"ContainerDied","Data":"d4e3a7a15dc6e345356703e67e93a6218ad11aacd744e0d49c4a07d2a5956949"} Oct 31 00:50:25.298903 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.298780 5011 scope.go:122] "RemoveContainer" containerID="6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519" Oct 31 00:50:25.300314 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.300287 5011 generic.go:356] "Generic (PLEG): container finished" podID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerID="b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19" exitCode=0 Oct 31 00:50:25.300419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.300335 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" Oct 31 00:50:25.300419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.300376 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" event={"ID":"18ac50da-c78d-489a-9fa1-686293c3ac84","Type":"ContainerDied","Data":"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19"} Oct 31 00:50:25.300522 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.300428 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf" event={"ID":"18ac50da-c78d-489a-9fa1-686293c3ac84","Type":"ContainerDied","Data":"0f092e69ab2e6f3d59b8f74ecbc5bf67e105995234526da7294dac640ca796bf"} Oct 31 00:50:25.311286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.311267 5011 scope.go:122] "RemoveContainer" containerID="6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519" Oct 31 00:50:25.311543 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:50:25.311524 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519\": container with ID starting with 6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519 not found: ID does not exist" containerID="6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519" Oct 31 00:50:25.311618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.311557 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519"} err="failed to get container status \"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519\": rpc error: code = NotFound desc = could not find container \"6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519\": container with ID starting with 6490ce56282303333aaeecac63a46d1743ff9f71daa82f9645e67de4e5b9a519 not found: ID does not exist" Oct 31 00:50:25.311618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.311582 5011 scope.go:122] "RemoveContainer" containerID="b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19" Oct 31 00:50:25.321463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.321445 5011 scope.go:122] "RemoveContainer" containerID="b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19" Oct 31 00:50:25.321764 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:50:25.321743 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19\": container with ID starting with b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19 not found: ID does not exist" containerID="b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19" Oct 31 00:50:25.321872 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.321769 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19"} err="failed to get container status \"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19\": rpc error: code = NotFound desc = could not find container \"b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19\": container with ID starting with b823c8a41411861052f9f1c4de7890bc12e394be87e3e28e816859b869b13b19 not found: ID does not exist" Oct 31 00:50:25.369611 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.369582 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:50:25.379478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.379453 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-nx8pf"] Oct 31 00:50:25.407761 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.407731 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:50:25.411914 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.411896 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-jc2v7"] Oct 31 00:50:25.685271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.685241 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" path="/var/lib/kubelet/pods/18ac50da-c78d-489a-9fa1-686293c3ac84/volumes" Oct 31 00:50:25.685813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:25.685795 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" path="/var/lib/kubelet/pods/2fd667b8-103b-44e9-b8b7-9f633ff7c4f0/volumes" Oct 31 00:50:32.113232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.112281 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.166258 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:50:32.165888 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:50:32.236231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.235330 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.602233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.593305 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.610687 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.609729 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.622249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.620719 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.645230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.634340 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.775237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:32.774368 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.098259 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.097309 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.242242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.241295 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 00:50:33.242242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.241784 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:50:33.610228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.609948 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.635229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.610518 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:50:33.639227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.636729 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.639227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.637253 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:50:33.701974 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.701919 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.902414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:33.902291 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 00:50:34.100609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.099174 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 00:50:34.100609 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.099250 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 00:50:34.101654 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.101628 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 00:50:34.282529 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.282449 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd"] Oct 31 00:50:34.283845 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.283820 5011 cpu_manager.go:414] "containerMap: removing container" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" Oct 31 00:50:34.283982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.283971 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" Oct 31 00:50:34.284062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284055 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" containerName="maas-api" Oct 31 00:50:34.284112 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284105 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" containerName="maas-api" Oct 31 00:50:34.284165 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284159 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" Oct 31 00:50:34.284237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284226 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" Oct 31 00:50:34.284481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284462 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d567ec-9a8c-458a-91b9-b7c61fb8ca60" containerName="maas-api" Oct 31 00:50:34.284577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284484 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd667b8-103b-44e9-b8b7-9f633ff7c4f0" containerName="istio-proxy" Oct 31 00:50:34.284577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.284507 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ac50da-c78d-489a-9fa1-686293c3ac84" containerName="istio-proxy" Oct 31 00:50:34.290092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.290071 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.306266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.305512 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:34.306266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.305527 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:50:34.308988 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.308966 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd"] Oct 31 00:50:34.369471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.369438 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcp8g\" (UniqueName: \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.369643 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.369481 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.369722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.369649 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.487191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.486402 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcp8g\" (UniqueName: \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.487191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.486453 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.487191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.486700 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.489474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.489422 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.502862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.502839 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.512695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.512661 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcp8g\" (UniqueName: \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\") pod \"maas-api-revoke-keys-e2e-api-a-14e039-nz2bd\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.610493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.610462 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:34.992244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:34.990251 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd"] Oct 31 00:50:35.896560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:35.896523 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" event={"ID":"3594cf51-4bad-4297-ba9a-ee77e9c0117e","Type":"ContainerStarted","Data":"0b6efc0021b11a77bf39e7041dc221b8ac179b8aa91bc34bb4662f3112bb627f"} Oct 31 00:50:35.897032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:35.896604 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" event={"ID":"3594cf51-4bad-4297-ba9a-ee77e9c0117e","Type":"ContainerStarted","Data":"ec066d7c5dedab2efd7608f194e454a1e8667e09218a2aecdefb00294f8e7160"} Oct 31 00:50:35.956660 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:35.956609 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" podStartSLOduration=1.9565938649999999 podStartE2EDuration="1.956593865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:35.947532869 +0000 UTC m=+1904.396620509" watchObservedRunningTime="2026-10-31 00:50:35.956593865 +0000 UTC m=+1904.405681504" Oct 31 00:50:36.906275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:36.906241 5011 generic.go:356] "Generic (PLEG): container finished" podID="3594cf51-4bad-4297-ba9a-ee77e9c0117e" containerID="0b6efc0021b11a77bf39e7041dc221b8ac179b8aa91bc34bb4662f3112bb627f" exitCode=0 Oct 31 00:50:36.906772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:36.906304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" event={"ID":"3594cf51-4bad-4297-ba9a-ee77e9c0117e","Type":"ContainerDied","Data":"0b6efc0021b11a77bf39e7041dc221b8ac179b8aa91bc34bb4662f3112bb627f"} Oct 31 00:50:38.101000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.100970 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:38.161464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.161426 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\" (UniqueName: \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\") pod \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " Oct 31 00:50:38.161722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.161701 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\") pod \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " Oct 31 00:50:38.161801 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.161767 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\") pod \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\" (UID: \"3594cf51-4bad-4297-ba9a-ee77e9c0117e\") " Oct 31 00:50:38.163437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.163389 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca" pod "3594cf51-4bad-4297-ba9a-ee77e9c0117e" (UID: "3594cf51-4bad-4297-ba9a-ee77e9c0117e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:38.165376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.165332 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g" pod "3594cf51-4bad-4297-ba9a-ee77e9c0117e" (UID: "3594cf51-4bad-4297-ba9a-ee77e9c0117e"). InnerVolumeSpecName "kube-api-access-jcp8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:38.184679 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.184652 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca" pod "3594cf51-4bad-4297-ba9a-ee77e9c0117e" (UID: "3594cf51-4bad-4297-ba9a-ee77e9c0117e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:38.264295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.264263 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:38.264295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.264296 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcp8g\" (UniqueName: \"kubernetes.io/projected/3594cf51-4bad-4297-ba9a-ee77e9c0117e-kube-api-access-jcp8g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:38.264489 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.264314 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3594cf51-4bad-4297-ba9a-ee77e9c0117e-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:38.928283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.928245 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" event={"ID":"3594cf51-4bad-4297-ba9a-ee77e9c0117e","Type":"ContainerDied","Data":"ec066d7c5dedab2efd7608f194e454a1e8667e09218a2aecdefb00294f8e7160"} Oct 31 00:50:38.928458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.928303 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec066d7c5dedab2efd7608f194e454a1e8667e09218a2aecdefb00294f8e7160" Oct 31 00:50:38.928458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:38.928305 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd" Oct 31 00:50:41.985003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.984966 5011 generic.go:356] "Generic (PLEG): container finished" podID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerID="eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4" exitCode=0 Oct 31 00:50:41.985570 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.985046 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerDied","Data":"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4"} Oct 31 00:50:41.985653 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.985636 5011 scope.go:122] "RemoveContainer" containerID="eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4" Oct 31 00:50:41.987638 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.987609 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:50:41.987725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.987679 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:50:41.987725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:41.987711 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:50:42.693666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:42.693634 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:50:42.996022 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:42.995924 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerStarted","Data":"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3"} Oct 31 00:50:43.138186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:43.138147 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:50:43.139283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:43.139255 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:43.139421 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:43.139304 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:44.142227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:44.141358 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:44.142227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:44.141419 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:45.148230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:45.147682 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:45.148230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:45.147777 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:46.147139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:46.147101 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:46.147444 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:46.147414 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:47.144971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.144539 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:47.144971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.144610 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:47.799478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.799438 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb"] Oct 31 00:50:47.800282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.800259 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3594cf51-4bad-4297-ba9a-ee77e9c0117e" containerName="revoke-keys" Oct 31 00:50:47.800403 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.800283 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3594cf51-4bad-4297-ba9a-ee77e9c0117e" containerName="revoke-keys" Oct 31 00:50:47.800472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.800455 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3594cf51-4bad-4297-ba9a-ee77e9c0117e" containerName="revoke-keys" Oct 31 00:50:47.804852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.804831 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.850235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.850005 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:47.850235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.850141 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:50:47.852934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.852900 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb"] Oct 31 00:50:47.885680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.885650 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.886032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.886005 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.886138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.886052 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rr8\" (UniqueName: \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.987736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.987701 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.987917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.987744 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rr8\" (UniqueName: \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.987917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.987826 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.988506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.988473 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:47.990193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:47.990166 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:48.010637 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:48.010612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rr8\" (UniqueName: \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\") pod \"maas-api-revoke-keys-e2e-api-a-e166ca-s44sb\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:48.143583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:48.143545 5011 patch_prober.go:29] interesting pod/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:50:48.143870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:48.143842 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 00:50:48.144561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:48.144538 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:48.532105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:48.532069 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb"] Oct 31 00:50:49.080948 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.080903 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" event={"ID":"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980","Type":"ContainerStarted","Data":"78fb5abcf9ed77f3fa8b1eb669ee1d09959be596acaeeae7ad41e34d6afee320"} Oct 31 00:50:49.081123 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.081007 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" event={"ID":"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980","Type":"ContainerStarted","Data":"cc3cd4ce4b452275cdd8927d25c6b5af8a000d1912a0f7dc7be9b4f687fa99f5"} Oct 31 00:50:49.137929 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.137706 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" podStartSLOduration=2.137687406 podStartE2EDuration="2.137687406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:49.125756706 +0000 UTC m=+1917.574844345" watchObservedRunningTime="2026-10-31 00:50:49.137687406 +0000 UTC m=+1917.586775046" Oct 31 00:50:49.225618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.225579 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:50:49.226336 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.226295 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:50:49.227634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:49.227566 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:50:50.091594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:50.091560 5011 generic.go:356] "Generic (PLEG): container finished" podID="5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" containerID="78fb5abcf9ed77f3fa8b1eb669ee1d09959be596acaeeae7ad41e34d6afee320" exitCode=0 Oct 31 00:50:50.092168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:50.091646 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" event={"ID":"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980","Type":"ContainerDied","Data":"78fb5abcf9ed77f3fa8b1eb669ee1d09959be596acaeeae7ad41e34d6afee320"} Oct 31 00:50:51.300055 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.300031 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:51.365692 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.365600 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\" (UniqueName: \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\") pod \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " Oct 31 00:50:51.366035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.365983 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\") pod \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " Oct 31 00:50:51.366313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.366276 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\") pod \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\" (UID: \"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980\") " Oct 31 00:50:51.366436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.366361 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca" pod "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" (UID: "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:51.367472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.367444 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:51.368186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.368163 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8" pod "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" (UID: "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980"). InnerVolumeSpecName "kube-api-access-g8rr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:51.369847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.369823 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca" pod "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" (UID: "5a18f8d5-90a2-4dd4-92e2-9ced72cbf980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:51.469163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.469129 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rr8\" (UniqueName: \"kubernetes.io/projected/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-kube-api-access-g8rr8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:51.469163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:51.469159 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:50:52.112954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.112914 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" event={"ID":"5a18f8d5-90a2-4dd4-92e2-9ced72cbf980","Type":"ContainerDied","Data":"cc3cd4ce4b452275cdd8927d25c6b5af8a000d1912a0f7dc7be9b4f687fa99f5"} Oct 31 00:50:52.113130 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.112981 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc3cd4ce4b452275cdd8927d25c6b5af8a000d1912a0f7dc7be9b4f687fa99f5" Oct 31 00:50:52.113130 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.113044 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb" Oct 31 00:50:52.571478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.571448 5011 scope.go:122] "RemoveContainer" containerID="2d155a06ca4ed97d0aa323cd00d50b4aa5cdd16276207866169662cfb8fbee8c" Oct 31 00:50:52.583046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.583006 5011 scope.go:122] "RemoveContainer" containerID="30f405b6b82e51d75f5c6904c5cb2482b581692fd3033fc3c3623ec1d1bc3c77" Oct 31 00:50:52.593579 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.593560 5011 scope.go:122] "RemoveContainer" containerID="475bc9f04e487980431cf392e160fed7ac38ba107efff72a51edc9c503710123" Oct 31 00:50:52.602746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.602729 5011 scope.go:122] "RemoveContainer" containerID="428fe6728c19102c74d63b3218c23fbf40a805e26fdf054817c3b89f836d26a1" Oct 31 00:50:52.611745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.611729 5011 scope.go:122] "RemoveContainer" containerID="fbd5bd482ab51110994af28af52191795f3f8b34c5e71336181d4a01d425b441" Oct 31 00:50:52.622159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.622136 5011 scope.go:122] "RemoveContainer" containerID="da42ce9bd839a4d1da2595f30c3a18533dbcced9642cfb813db591b104ebc20e" Oct 31 00:50:52.632730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:52.632710 5011 scope.go:122] "RemoveContainer" containerID="5b53bfcf3b160f3546bab04a37299242950f04bed0c3cc24a81fa34963998dfa" Oct 31 00:50:55.470159 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:55.470127 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:50:56.882819 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.882789 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns"] Oct 31 00:50:56.883574 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.883550 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" containerName="revoke-keys" Oct 31 00:50:56.883671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.883576 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" containerName="revoke-keys" Oct 31 00:50:56.883839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.883826 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" containerName="revoke-keys" Oct 31 00:50:56.890333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.890312 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:56.907482 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.907456 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns"] Oct 31 00:50:56.918268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.913114 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:56.918268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.913153 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:50:56.918759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.918729 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:56.918973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.918953 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pzg\" (UniqueName: \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:56.919054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:56.919039 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.056275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.055133 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pzg\" (UniqueName: \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.056275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.055256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.056275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.055379 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.060227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.057092 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.148333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.101448 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.148333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.127882 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pzg\" (UniqueName: \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\") pod \"maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.228897 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.228850 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:50:57.579708 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:57.579602 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns"] Oct 31 00:50:57.581662 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:50:57.580960 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5208dc_26da_49bd_8a1d_f118fb1279e1.slice/crio-a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7 WatchSource:0}: Error finding container a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7: Status 404 returned error can't find the container with id a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7 Oct 31 00:50:58.247178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:58.247083 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" event={"ID":"4b5208dc-26da-49bd-8a1d-f118fb1279e1","Type":"ContainerStarted","Data":"259659bfac94e4708774cf11fa423a3cd281a3ee4907ced1016ad9b79513a98f"} Oct 31 00:50:58.247178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:58.247153 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" event={"ID":"4b5208dc-26da-49bd-8a1d-f118fb1279e1","Type":"ContainerStarted","Data":"a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7"} Oct 31 00:50:59.256795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:59.256758 5011 generic.go:356] "Generic (PLEG): container finished" podID="4b5208dc-26da-49bd-8a1d-f118fb1279e1" containerID="259659bfac94e4708774cf11fa423a3cd281a3ee4907ced1016ad9b79513a98f" exitCode=0 Oct 31 00:50:59.257178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:50:59.256840 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" event={"ID":"4b5208dc-26da-49bd-8a1d-f118fb1279e1","Type":"ContainerDied","Data":"259659bfac94e4708774cf11fa423a3cd281a3ee4907ced1016ad9b79513a98f"} Oct 31 00:51:00.439086 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.439046 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:51:00.618304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.618266 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\" (UniqueName: \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\") pod \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " Oct 31 00:51:00.618493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.618379 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\") pod \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " Oct 31 00:51:00.618493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.618466 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\") pod \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\" (UID: \"4b5208dc-26da-49bd-8a1d-f118fb1279e1\") " Oct 31 00:51:00.618786 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.618757 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca" pod "4b5208dc-26da-49bd-8a1d-f118fb1279e1" (UID: "4b5208dc-26da-49bd-8a1d-f118fb1279e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:00.619178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.619155 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:00.620671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.620644 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca" pod "4b5208dc-26da-49bd-8a1d-f118fb1279e1" (UID: "4b5208dc-26da-49bd-8a1d-f118fb1279e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:00.620746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.620686 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg" pod "4b5208dc-26da-49bd-8a1d-f118fb1279e1" (UID: "4b5208dc-26da-49bd-8a1d-f118fb1279e1"). InnerVolumeSpecName "kube-api-access-k5pzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:00.720463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.720434 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5pzg\" (UniqueName: \"kubernetes.io/projected/4b5208dc-26da-49bd-8a1d-f118fb1279e1-kube-api-access-k5pzg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:00.720463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:00.720462 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b5208dc-26da-49bd-8a1d-f118fb1279e1-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:01.274729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:01.274699 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" Oct 31 00:51:01.274729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:01.274708 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns" event={"ID":"4b5208dc-26da-49bd-8a1d-f118fb1279e1","Type":"ContainerDied","Data":"a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7"} Oct 31 00:51:01.274952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:01.274759 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a206b27644a5d116d37aa39106745cf8b818835e2ef68b1b13105ec5263375b7" Oct 31 00:51:01.948343 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:01.948307 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:51:04.751754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:04.751665 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:51:04.752242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:04.751959 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" containerID="cri-o://33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab" gracePeriod=30 Oct 31 00:51:09.648405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:09.648361 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns"] Oct 31 00:51:09.669097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:09.669058 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2c15c1-8dhns"] Oct 31 00:51:09.700730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:09.700700 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5208dc-26da-49bd-8a1d-f118fb1279e1" path="/var/lib/kubelet/pods/4b5208dc-26da-49bd-8a1d-f118fb1279e1/volumes" Oct 31 00:51:10.054173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.054143 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:51:10.096097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096066 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096135 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096278 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096353 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096385 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096413 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096478 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096524 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.096663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.096613 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\") pod \"08051a91-99a7-4823-8150-24e545e2e4f3\" (UID: \"08051a91-99a7-4823-8150-24e545e2e4f3\") " Oct 31 00:51:10.097453 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.097424 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:10.097618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.097602 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:10.097892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.097792 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:10.097977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.097922 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:10.098222 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.098066 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:10.100459 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.100428 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:10.100647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.100622 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:10.100728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.100699 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:10.101343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.101321 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh" pod "08051a91-99a7-4823-8150-24e545e2e4f3" (UID: "08051a91-99a7-4823-8150-24e545e2e4f3"). InnerVolumeSpecName "kube-api-access-kmhhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198749 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198779 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198789 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198798 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmhhh\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-kube-api-access-kmhhh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198809 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08051a91-99a7-4823-8150-24e545e2e4f3-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198818 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.198821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198826 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08051a91-99a7-4823-8150-24e545e2e4f3-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.199143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198836 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08051a91-99a7-4823-8150-24e545e2e4f3-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.199143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.198848 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08051a91-99a7-4823-8150-24e545e2e4f3-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:10.593360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.593327 5011 generic.go:356] "Generic (PLEG): container finished" podID="08051a91-99a7-4823-8150-24e545e2e4f3" containerID="33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab" exitCode=0 Oct 31 00:51:10.593533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.593392 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" Oct 31 00:51:10.593533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.593411 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" event={"ID":"08051a91-99a7-4823-8150-24e545e2e4f3","Type":"ContainerDied","Data":"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab"} Oct 31 00:51:10.593533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.593479 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c" event={"ID":"08051a91-99a7-4823-8150-24e545e2e4f3","Type":"ContainerDied","Data":"d9a79ccdb5a518c77807e5b4855ba8b5af9bc847906bd4ea406e573c7f24960f"} Oct 31 00:51:10.593533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.593509 5011 scope.go:122] "RemoveContainer" containerID="33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab" Oct 31 00:51:10.605196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.605176 5011 scope.go:122] "RemoveContainer" containerID="33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab" Oct 31 00:51:10.605508 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:10.605487 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab\": container with ID starting with 33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab not found: ID does not exist" containerID="33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab" Oct 31 00:51:10.605594 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.605517 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab"} err="failed to get container status \"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab\": rpc error: code = NotFound desc = could not find container \"33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab\": container with ID starting with 33a05fba4f7cd224e47a759b82c2f9888c3aae7fe666063db3deead70a6253ab not found: ID does not exist" Oct 31 00:51:10.647698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.647675 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:51:10.659691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:10.659669 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zhd2c"] Oct 31 00:51:11.687466 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:11.687430 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" path="/var/lib/kubelet/pods/08051a91-99a7-4823-8150-24e545e2e4f3/volumes" Oct 31 00:51:19.486772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:19.486717 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:51:19.488538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:19.488486 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" containerID="cri-o://cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e" gracePeriod=30 Oct 31 00:51:24.787146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.787117 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:51:24.904974 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.904939 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.905587 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.905568 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.905957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.905941 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.906094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.906081 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.906284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.906268 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.906423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.906408 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.922768 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.905465 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.922973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.922731 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:24.923400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.923378 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.923746 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.923728 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.924355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.924335 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.924627 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.924599 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.938257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.932378 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\") pod \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\" (UID: \"657f029d-a7b4-4fc7-844b-6a74bf714ce5\") " Oct 31 00:51:24.938257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.933508 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:24.938257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.933533 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:24.938257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.933550 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:24.938257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.933564 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:24.948635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:24.948579 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:25.070399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.070330 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:25.070605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.070499 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "kube-api-access-q2r5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:25.070605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.070564 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:25.096167 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.095889 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo" pod "657f029d-a7b4-4fc7-844b-6a74bf714ce5" (UID: "657f029d-a7b4-4fc7-844b-6a74bf714ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:25.125002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.124964 5011 generic.go:356] "Generic (PLEG): container finished" podID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerID="cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e" exitCode=0 Oct 31 00:51:25.125497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.125469 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerDied","Data":"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e"} Oct 31 00:51:25.127272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.127245 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" event={"ID":"657f029d-a7b4-4fc7-844b-6a74bf714ce5","Type":"ContainerDied","Data":"7596bfd4e8553e6aa50ba52acc102f216f2099ffe4acb6d66c5c8d2e5415204f"} Oct 31 00:51:25.127420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.127398 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8" Oct 31 00:51:25.127550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.127533 5011 scope.go:122] "RemoveContainer" containerID="cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e" Oct 31 00:51:25.132632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.132605 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2r5p\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-kube-api-access-q2r5p\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:25.132751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.132646 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:25.132751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.132663 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:25.132751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.132687 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:25.132751 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.132703 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657f029d-a7b4-4fc7-844b-6a74bf714ce5-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:25.143033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.143010 5011 scope.go:122] "RemoveContainer" containerID="5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79" Oct 31 00:51:25.154418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.154390 5011 scope.go:122] "RemoveContainer" containerID="cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e" Oct 31 00:51:25.154696 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:25.154669 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e\": container with ID starting with cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e not found: ID does not exist" containerID="cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e" Oct 31 00:51:25.154763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.154708 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e"} err="failed to get container status \"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e\": rpc error: code = NotFound desc = could not find container \"cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e\": container with ID starting with cbe36f0ea70be6ba3cd03f452f8754deb91850bcf912cac757316b6ee829229e not found: ID does not exist" Oct 31 00:51:25.154763 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.154732 5011 scope.go:122] "RemoveContainer" containerID="5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79" Oct 31 00:51:25.155019 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:25.154995 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79\": container with ID starting with 5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79 not found: ID does not exist" containerID="5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79" Oct 31 00:51:25.155080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.155030 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79"} err="failed to get container status \"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79\": rpc error: code = NotFound desc = could not find container \"5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79\": container with ID starting with 5a579b496bc885b9477c3bd9f67fbeb8474589ede05543e875c80af6cb4f3e79 not found: ID does not exist" Oct 31 00:51:25.185023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.185001 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:51:25.196246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.196177 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fa2c15c1-openshift-default-7d89b995b9-829k8"] Oct 31 00:51:25.685073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:25.685039 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" path="/var/lib/kubelet/pods/657f029d-a7b4-4fc7-844b-6a74bf714ce5/volumes" Oct 31 00:51:28.966704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:28.966672 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:51:28.967298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:28.966948 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" podUID="8a79c53d-2463-4aef-8d4d-f376f978ab80" containerName="maas-api" containerID="cri-o://e4c76d89b0c5fc87eadc79159541028d3dc98e21b3ece6335978feb87dd85a42" gracePeriod=30 Oct 31 00:51:29.168773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.168733 5011 generic.go:356] "Generic (PLEG): container finished" podID="8a79c53d-2463-4aef-8d4d-f376f978ab80" containerID="e4c76d89b0c5fc87eadc79159541028d3dc98e21b3ece6335978feb87dd85a42" exitCode=0 Oct 31 00:51:29.168990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.168973 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" event={"ID":"8a79c53d-2463-4aef-8d4d-f376f978ab80","Type":"ContainerDied","Data":"e4c76d89b0c5fc87eadc79159541028d3dc98e21b3ece6335978feb87dd85a42"} Oct 31 00:51:29.849343 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.849312 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:51:29.948558 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.948524 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd"] Oct 31 00:51:29.952958 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.952932 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\") pod \"8a79c53d-2463-4aef-8d4d-f376f978ab80\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " Oct 31 00:51:29.953080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.952981 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\") pod \"8a79c53d-2463-4aef-8d4d-f376f978ab80\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " Oct 31 00:51:29.953129 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.953080 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\" (UniqueName: \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\") pod \"8a79c53d-2463-4aef-8d4d-f376f978ab80\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " Oct 31 00:51:29.953279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.953247 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\") pod \"8a79c53d-2463-4aef-8d4d-f376f978ab80\" (UID: \"8a79c53d-2463-4aef-8d4d-f376f978ab80\") " Oct 31 00:51:29.953722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.953695 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca" pod "8a79c53d-2463-4aef-8d4d-f376f978ab80" (UID: "8a79c53d-2463-4aef-8d4d-f376f978ab80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:29.954182 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.954151 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a79c53d-2463-4aef-8d4d-f376f978ab80-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:29.954947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.954921 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14e039-nz2bd"] Oct 31 00:51:29.955595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.955568 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh" pod "8a79c53d-2463-4aef-8d4d-f376f978ab80" (UID: "8a79c53d-2463-4aef-8d4d-f376f978ab80"). InnerVolumeSpecName "kube-api-access-5k6wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:29.955683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.955568 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls" pod "8a79c53d-2463-4aef-8d4d-f376f978ab80" (UID: "8a79c53d-2463-4aef-8d4d-f376f978ab80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:29.955811 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:29.955789 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs" pod "8a79c53d-2463-4aef-8d4d-f376f978ab80" (UID: "8a79c53d-2463-4aef-8d4d-f376f978ab80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:30.055838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.055796 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:30.055838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.055838 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a79c53d-2463-4aef-8d4d-f376f978ab80-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:30.056404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.055853 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6wh\" (UniqueName: \"kubernetes.io/projected/8a79c53d-2463-4aef-8d4d-f376f978ab80-kube-api-access-5k6wh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:30.202779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.202612 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" event={"ID":"8a79c53d-2463-4aef-8d4d-f376f978ab80","Type":"ContainerDied","Data":"ef95e39d2dc941f3f1bf0ab152b517002b73a7a34e2a6170c379b3f3cae4b018"} Oct 31 00:51:30.202779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.202693 5011 scope.go:122] "RemoveContainer" containerID="e4c76d89b0c5fc87eadc79159541028d3dc98e21b3ece6335978feb87dd85a42" Oct 31 00:51:30.203013 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.202849 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw" Oct 31 00:51:30.267187 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.267142 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:51:30.272979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:30.272946 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e039-58d55fdf65-tqzvw"] Oct 31 00:51:31.688365 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:31.688323 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3594cf51-4bad-4297-ba9a-ee77e9c0117e" path="/var/lib/kubelet/pods/3594cf51-4bad-4297-ba9a-ee77e9c0117e/volumes" Oct 31 00:51:31.688985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:31.688963 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a79c53d-2463-4aef-8d4d-f376f978ab80" path="/var/lib/kubelet/pods/8a79c53d-2463-4aef-8d4d-f376f978ab80/volumes" Oct 31 00:51:31.962486 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:31.962396 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:51:38.761915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:38.761781 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:51:38.762794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:38.762738 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" containerID="cri-o://1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028" gracePeriod=30 Oct 31 00:51:39.377770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.377725 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp"] Oct 31 00:51:39.378449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378427 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8a79c53d-2463-4aef-8d4d-f376f978ab80" containerName="maas-api" Oct 31 00:51:39.378449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378443 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a79c53d-2463-4aef-8d4d-f376f978ab80" containerName="maas-api" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378455 5011 cpu_manager.go:414] "containerMap: removing container" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378460 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378471 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4b5208dc-26da-49bd-8a1d-f118fb1279e1" containerName="revoke-keys" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378477 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5208dc-26da-49bd-8a1d-f118fb1279e1" containerName="revoke-keys" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378486 5011 cpu_manager.go:414] "containerMap: removing container" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378490 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378519 5011 cpu_manager.go:414] "containerMap: removing container" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" Oct 31 00:51:39.378622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378523 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" Oct 31 00:51:39.378870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378641 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="08051a91-99a7-4823-8150-24e545e2e4f3" containerName="istio-proxy" Oct 31 00:51:39.378870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378656 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a79c53d-2463-4aef-8d4d-f376f978ab80" containerName="maas-api" Oct 31 00:51:39.378870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378663 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:39.378870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.378669 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5208dc-26da-49bd-8a1d-f118fb1279e1" containerName="revoke-keys" Oct 31 00:51:39.383651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.383631 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.401005 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.400979 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp"] Oct 31 00:51:39.407745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.407721 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:39.407745 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.407731 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:51:39.481335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.481291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6c8x\" (UniqueName: \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.481534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.481351 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.481606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.481578 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.582605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.582563 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.582807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.582794 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6c8x\" (UniqueName: \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.582881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.582830 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.583299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.583270 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.585436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.585416 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.613198 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.613164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6c8x\" (UniqueName: \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\") pod \"maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:39.700098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:39.700018 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:40.052046 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:51:40.052016 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd47b3ef_f9b7_432a_ad76_1a1a42b0bc5a.slice/crio-badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460 WatchSource:0}: Error finding container badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460: Status 404 returned error can't find the container with id badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460 Oct 31 00:51:40.055268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:40.054620 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp"] Oct 31 00:51:40.504146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:40.504106 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" event={"ID":"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a","Type":"ContainerStarted","Data":"d7a953c7ac3c848070aad4369c2ad9dc47006f61fbacd42fd766f26b2eca96af"} Oct 31 00:51:40.504328 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:40.504174 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" event={"ID":"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a","Type":"ContainerStarted","Data":"badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460"} Oct 31 00:51:40.547412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:40.547361 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" podStartSLOduration=1.5473438160000001 podStartE2EDuration="1.547343816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:40.544742186 +0000 UTC m=+1968.993829839" watchObservedRunningTime="2026-10-31 00:51:40.547343816 +0000 UTC m=+1968.996431456" Oct 31 00:51:41.514015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:41.513985 5011 generic.go:356] "Generic (PLEG): container finished" podID="fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" containerID="d7a953c7ac3c848070aad4369c2ad9dc47006f61fbacd42fd766f26b2eca96af" exitCode=0 Oct 31 00:51:41.514439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:41.514030 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" event={"ID":"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a","Type":"ContainerDied","Data":"d7a953c7ac3c848070aad4369c2ad9dc47006f61fbacd42fd766f26b2eca96af"} Oct 31 00:51:42.755161 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.755132 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:42.844891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.844854 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\" (UniqueName: \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\") pod \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " Oct 31 00:51:42.845097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.845076 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\") pod \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " Oct 31 00:51:42.845309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.845290 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\") pod \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\" (UID: \"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a\") " Oct 31 00:51:42.845699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.845666 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca" pod "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" (UID: "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:42.846002 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.845976 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:42.848318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.848280 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca" pod "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" (UID: "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:42.848318 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.848306 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x" pod "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" (UID: "fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a"). InnerVolumeSpecName "kube-api-access-v6c8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:42.948029 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.947996 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:42.948029 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:42.948029 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6c8x\" (UniqueName: \"kubernetes.io/projected/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a-kube-api-access-v6c8x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:43.539990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:43.539952 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" Oct 31 00:51:43.539990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:43.539968 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp" event={"ID":"fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a","Type":"ContainerDied","Data":"badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460"} Oct 31 00:51:43.540281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:43.540051 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="badaa7833c92876b1bfc1394f5586b4469583360a70ff3d16efb65eab37ae460" Oct 31 00:51:44.094716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.094689 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:51:44.173641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173604 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.173832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173694 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.173832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173810 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.173963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173886 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.173963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173937 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.174077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.173980 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.174077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.174014 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.174275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.174257 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.174359 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.174311 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\") pod \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\" (UID: \"3040c175-a485-4d7a-9b8b-60bcfa6340dd\") " Oct 31 00:51:44.174418 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.174349 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:44.175179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.175148 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.175422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.175400 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:44.176671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.175694 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:44.176767 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.176702 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:44.176930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.176892 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:44.177018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.176931 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:44.177082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.177045 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:44.177427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.177403 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:44.179325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.179305 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k" pod "3040c175-a485-4d7a-9b8b-60bcfa6340dd" (UID: "3040c175-a485-4d7a-9b8b-60bcfa6340dd"). InnerVolumeSpecName "kube-api-access-d2q7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:44.276772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276739 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276772 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276771 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276782 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276791 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276799 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276807 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3040c175-a485-4d7a-9b8b-60bcfa6340dd-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276815 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2q7k\" (UniqueName: \"kubernetes.io/projected/3040c175-a485-4d7a-9b8b-60bcfa6340dd-kube-api-access-d2q7k\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.276963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.276823 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3040c175-a485-4d7a-9b8b-60bcfa6340dd-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:44.552573 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.552528 5011 generic.go:356] "Generic (PLEG): container finished" podID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerID="1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028" exitCode=0 Oct 31 00:51:44.552748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.552588 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" Oct 31 00:51:44.552748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.552613 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerDied","Data":"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028"} Oct 31 00:51:44.552748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.552674 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls" event={"ID":"3040c175-a485-4d7a-9b8b-60bcfa6340dd","Type":"ContainerDied","Data":"e45df68407e4b1ca9ac6d7a8dfd14d9db477ff7e74ae33ddfae2b21ccf358e56"} Oct 31 00:51:44.552748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.552697 5011 scope.go:122] "RemoveContainer" containerID="1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028" Oct 31 00:51:44.565234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.565201 5011 scope.go:122] "RemoveContainer" containerID="d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7" Oct 31 00:51:44.575303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.575286 5011 scope.go:122] "RemoveContainer" containerID="1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028" Oct 31 00:51:44.575554 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:44.575534 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028\": container with ID starting with 1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028 not found: ID does not exist" containerID="1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028" Oct 31 00:51:44.575630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.575563 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028"} err="failed to get container status \"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028\": rpc error: code = NotFound desc = could not find container \"1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028\": container with ID starting with 1154ad92f3296e1c3050dce7d61eea5eddbb9483d636102df3b1fb1a41f69028 not found: ID does not exist" Oct 31 00:51:44.575630 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.575579 5011 scope.go:122] "RemoveContainer" containerID="d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7" Oct 31 00:51:44.575810 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:44.575796 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7\": container with ID starting with d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7 not found: ID does not exist" containerID="d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7" Oct 31 00:51:44.575848 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.575816 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7"} err="failed to get container status \"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7\": rpc error: code = NotFound desc = could not find container \"d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7\": container with ID starting with d3e34a875d331e55b935aa3c98cb7a1ef55ba576f7ba5e497cd33620b6ae89b7 not found: ID does not exist" Oct 31 00:51:44.607629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.607594 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:51:44.619430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:44.619406 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-14e039-openshift-default-5f7fc9955c-pcgls"] Oct 31 00:51:45.691248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:45.687442 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" path="/var/lib/kubelet/pods/3040c175-a485-4d7a-9b8b-60bcfa6340dd/volumes" Oct 31 00:51:45.886955 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:45.886917 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb"] Oct 31 00:51:45.899128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:45.899099 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e166ca-s44sb"] Oct 31 00:51:46.160650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.160614 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10896-bc4985"] Oct 31 00:51:46.161413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161387 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" containerName="revoke-keys" Oct 31 00:51:46.161413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161410 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" containerName="revoke-keys" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161423 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161428 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161439 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161444 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161612 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" containerName="revoke-keys" Oct 31 00:51:46.161635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161627 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161640 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3040c175-a485-4d7a-9b8b-60bcfa6340dd" containerName="istio-proxy" Oct 31 00:51:46.161963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.161650 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="657f029d-a7b4-4fc7-844b-6a74bf714ce5" containerName="istio-proxy" Oct 31 00:51:46.167088 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.167061 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:46.181975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.181952 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\"" Oct 31 00:51:46.183555 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.183530 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-bc4985"] Oct 31 00:51:46.221634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.221606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2z8f\" (UniqueName: \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\") pod \"test-curl-10896-bc4985\" (UID: \"1acf58f0-a9b8-4c46-9e50-828ec551c19a\") " pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:46.322646 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.322612 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2z8f\" (UniqueName: \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\") pod \"test-curl-10896-bc4985\" (UID: \"1acf58f0-a9b8-4c46-9e50-828ec551c19a\") " pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:46.346433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.346401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2z8f\" (UniqueName: \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\") pod \"test-curl-10896-bc4985\" (UID: \"1acf58f0-a9b8-4c46-9e50-828ec551c19a\") " pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:46.483683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.483608 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:46.796537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:46.796506 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-bc4985"] Oct 31 00:51:46.796998 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:51:46.796974 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1acf58f0_a9b8_4c46_9e50_828ec551c19a.slice/crio-d042b91f0d270b47e7b1c0538f31fb48f520e8a8e31b049df6724aef411f22da WatchSource:0}: Error finding container d042b91f0d270b47e7b1c0538f31fb48f520e8a8e31b049df6724aef411f22da: Status 404 returned error can't find the container with id d042b91f0d270b47e7b1c0538f31fb48f520e8a8e31b049df6724aef411f22da Oct 31 00:51:47.213703 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:47.213674 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:51:47.587082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:47.587046 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-bc4985" event={"ID":"1acf58f0-a9b8-4c46-9e50-828ec551c19a","Type":"ContainerStarted","Data":"dbe6a2d28ccb8098e05d2eb362324ff823108dbc4a8e270facd7b01fb76cc75e"} Oct 31 00:51:47.587262 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:47.587092 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-bc4985" event={"ID":"1acf58f0-a9b8-4c46-9e50-828ec551c19a","Type":"ContainerStarted","Data":"d042b91f0d270b47e7b1c0538f31fb48f520e8a8e31b049df6724aef411f22da"} Oct 31 00:51:47.640875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:47.640823 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10896-bc4985" podStartSLOduration=1.640810733 podStartE2EDuration="1.640810733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:47.635911533 +0000 UTC m=+1976.084999173" watchObservedRunningTime="2026-10-31 00:51:47.640810733 +0000 UTC m=+1976.089898371" Oct 31 00:51:47.756334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:47.756296 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a18f8d5-90a2-4dd4-92e2-9ced72cbf980" path="/var/lib/kubelet/pods/5a18f8d5-90a2-4dd4-92e2-9ced72cbf980/volumes" Oct 31 00:51:48.720790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:48.720758 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10896-bc4985"] Oct 31 00:51:48.728773 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:48.728744 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10896-bc4985"] Oct 31 00:51:49.028006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.027916 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq"] Oct 31 00:51:49.028887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.028862 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" containerName="test-curl-10896-bc4985" Oct 31 00:51:49.028887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.028886 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" containerName="test-curl-10896-bc4985" Oct 31 00:51:49.029049 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.029040 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" containerName="test-curl-10896-bc4985" Oct 31 00:51:49.033066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.033046 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.050461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.050435 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq"] Oct 31 00:51:49.055476 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.055450 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:49.055592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.055507 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:51:49.182503 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.182468 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.182673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.182617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlwg\" (UniqueName: \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.182673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.182647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.284467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.284393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.284629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.284502 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlwg\" (UniqueName: \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.284629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.284533 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.285031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.285002 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.286955 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.286935 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.308794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.308768 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlwg\" (UniqueName: \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\") pod \"maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.349072 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.349043 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:49.604274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.604239 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10896-bc4985" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" containerName="test-curl-10896-bc4985" containerID="cri-o://dbe6a2d28ccb8098e05d2eb362324ff823108dbc4a8e270facd7b01fb76cc75e" gracePeriod=2 Oct 31 00:51:49.618121 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:49.618091 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-bc4985\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:49.668892 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:51:49.668857 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782ae276_5bf4_4617_b01d_2882f2c36e71.slice/crio-136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92 WatchSource:0}: Error finding container 136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92: Status 404 returned error can't find the container with id 136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92 Oct 31 00:51:49.669251 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:49.669225 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq"] Oct 31 00:51:50.704231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:50.703292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" event={"ID":"782ae276-5bf4-4617-b01d-2882f2c36e71","Type":"ContainerStarted","Data":"8de09a2e3fc3ce7ed02c6ffce94baaae209131ad0cd7d2fc6547a7a2c8cefb72"} Oct 31 00:51:50.704231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:50.703354 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" event={"ID":"782ae276-5bf4-4617-b01d-2882f2c36e71","Type":"ContainerStarted","Data":"136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92"} Oct 31 00:51:51.714345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.714313 5011 generic.go:356] "Generic (PLEG): container finished" podID="782ae276-5bf4-4617-b01d-2882f2c36e71" containerID="8de09a2e3fc3ce7ed02c6ffce94baaae209131ad0cd7d2fc6547a7a2c8cefb72" exitCode=0 Oct 31 00:51:51.714778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.714394 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" event={"ID":"782ae276-5bf4-4617-b01d-2882f2c36e71","Type":"ContainerDied","Data":"8de09a2e3fc3ce7ed02c6ffce94baaae209131ad0cd7d2fc6547a7a2c8cefb72"} Oct 31 00:51:51.716102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.716076 5011 generic.go:356] "Generic (PLEG): container finished" podID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" containerID="dbe6a2d28ccb8098e05d2eb362324ff823108dbc4a8e270facd7b01fb76cc75e" exitCode=137 Oct 31 00:51:51.891250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.891224 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:51.905536 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:51.905490 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-bc4985\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:51.980409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.980378 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\" (UniqueName: \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\") pod \"1acf58f0-a9b8-4c46-9e50-828ec551c19a\" (UID: \"1acf58f0-a9b8-4c46-9e50-828ec551c19a\") " Oct 31 00:51:51.984269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.983222 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f" pod "1acf58f0-a9b8-4c46-9e50-828ec551c19a" (UID: "1acf58f0-a9b8-4c46-9e50-828ec551c19a"). InnerVolumeSpecName "kube-api-access-v2z8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:51.990436 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:51.990406 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:52.047544 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:52.047501 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-bc4985\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:52.082340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.082306 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\" (UniqueName: \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\") pod \"782ae276-5bf4-4617-b01d-2882f2c36e71\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " Oct 31 00:51:52.082504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.082454 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\") pod \"782ae276-5bf4-4617-b01d-2882f2c36e71\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " Oct 31 00:51:52.082605 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.082587 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\") pod \"782ae276-5bf4-4617-b01d-2882f2c36e71\" (UID: \"782ae276-5bf4-4617-b01d-2882f2c36e71\") " Oct 31 00:51:52.083190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.083170 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2z8f\" (UniqueName: \"kubernetes.io/projected/1acf58f0-a9b8-4c46-9e50-828ec551c19a-kube-api-access-v2z8f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:52.083350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.083236 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca" pod "782ae276-5bf4-4617-b01d-2882f2c36e71" (UID: "782ae276-5bf4-4617-b01d-2882f2c36e71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:52.085178 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.085148 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg" pod "782ae276-5bf4-4617-b01d-2882f2c36e71" (UID: "782ae276-5bf4-4617-b01d-2882f2c36e71"). InnerVolumeSpecName "kube-api-access-6jlwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:52.085294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.085253 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca" pod "782ae276-5bf4-4617-b01d-2882f2c36e71" (UID: "782ae276-5bf4-4617-b01d-2882f2c36e71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:52.184546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.184470 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:52.184546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.184496 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782ae276-5bf4-4617-b01d-2882f2c36e71-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:52.184546 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.184506 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlwg\" (UniqueName: \"kubernetes.io/projected/782ae276-5bf4-4617-b01d-2882f2c36e71-kube-api-access-6jlwg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:52.712115 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.712082 5011 scope.go:122] "RemoveContainer" containerID="b86adb55b67fc28720cffffb3594836af9e03fdb6a266a31cd0adda07f02266c" Oct 31 00:51:52.725053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.725023 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:52.725433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.725072 5011 scope.go:122] "RemoveContainer" containerID="dbe6a2d28ccb8098e05d2eb362324ff823108dbc4a8e270facd7b01fb76cc75e" Oct 31 00:51:52.727059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.727039 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" Oct 31 00:51:52.727172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.727041 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq" event={"ID":"782ae276-5bf4-4617-b01d-2882f2c36e71","Type":"ContainerDied","Data":"136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92"} Oct 31 00:51:52.727248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:52.727194 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="136ef62983ae794d3933a785247afc21c8f59f3b9de0b5448058804ba9c9dd92" Oct 31 00:51:52.746541 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:52.746506 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-bc4985\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:52.775012 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:51:52.774986 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-bc4985\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" pod="opendatahub/test-curl-10896-bc4985" Oct 31 00:51:53.686018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:53.685980 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acf58f0-a9b8-4c46-9e50-828ec551c19a" path="/var/lib/kubelet/pods/1acf58f0-a9b8-4c46-9e50-828ec551c19a/volumes" Oct 31 00:51:54.487524 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.487483 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr"] Oct 31 00:51:54.489790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.489766 5011 cpu_manager.go:414] "containerMap: removing container" podUID="782ae276-5bf4-4617-b01d-2882f2c36e71" containerName="revoke-keys" Oct 31 00:51:54.489956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.489941 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782ae276-5bf4-4617-b01d-2882f2c36e71" containerName="revoke-keys" Oct 31 00:51:54.491239 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.491218 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="782ae276-5bf4-4617-b01d-2882f2c36e71" containerName="revoke-keys" Oct 31 00:51:54.499946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.499919 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.539235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.533015 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.539235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.533088 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.539235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.533268 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzkz\" (UniqueName: \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.539235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.538755 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr"] Oct 31 00:51:54.580559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.578499 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:54.580559 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.578648 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:51:54.637278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.634970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.637278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.635012 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.637278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.635087 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzkz\" (UniqueName: \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.637278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.636295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.638432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.638404 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.658519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.658489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzkz\" (UniqueName: \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\") pod \"maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:54.874229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:54.871481 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:55.234182 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:51:55.234148 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59338868_c02c_4f78_bbc6_e45e09bb6bdb.slice/crio-e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547 WatchSource:0}: Error finding container e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547: Status 404 returned error can't find the container with id e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547 Oct 31 00:51:55.235521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:55.235494 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr"] Oct 31 00:51:55.764414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:55.764377 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" event={"ID":"59338868-c02c-4f78-bbc6-e45e09bb6bdb","Type":"ContainerStarted","Data":"f97d89e6e00690cbe6bdaf6b538185c8d531d92f98f040685423306e94b76998"} Oct 31 00:51:55.764822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:55.764439 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" event={"ID":"59338868-c02c-4f78-bbc6-e45e09bb6bdb","Type":"ContainerStarted","Data":"e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547"} Oct 31 00:51:55.808518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:55.808471 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" podStartSLOduration=1.808455151 podStartE2EDuration="1.808455151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:55.805242483 +0000 UTC m=+1984.254330122" watchObservedRunningTime="2026-10-31 00:51:55.808455151 +0000 UTC m=+1984.257542790" Oct 31 00:51:56.774341 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:56.774295 5011 generic.go:356] "Generic (PLEG): container finished" podID="59338868-c02c-4f78-bbc6-e45e09bb6bdb" containerID="f97d89e6e00690cbe6bdaf6b538185c8d531d92f98f040685423306e94b76998" exitCode=0 Oct 31 00:51:56.774750 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:56.774376 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" event={"ID":"59338868-c02c-4f78-bbc6-e45e09bb6bdb","Type":"ContainerDied","Data":"f97d89e6e00690cbe6bdaf6b538185c8d531d92f98f040685423306e94b76998"} Oct 31 00:51:57.975350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:57.975314 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:58.115591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.115558 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\") pod \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " Oct 31 00:51:58.115842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.115824 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\" (UniqueName: \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\") pod \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " Oct 31 00:51:58.115926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.115895 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\") pod \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\" (UID: \"59338868-c02c-4f78-bbc6-e45e09bb6bdb\") " Oct 31 00:51:58.116407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.116370 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca" pod "59338868-c02c-4f78-bbc6-e45e09bb6bdb" (UID: "59338868-c02c-4f78-bbc6-e45e09bb6bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:58.116565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.116545 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:58.117988 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.117960 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz" pod "59338868-c02c-4f78-bbc6-e45e09bb6bdb" (UID: "59338868-c02c-4f78-bbc6-e45e09bb6bdb"). InnerVolumeSpecName "kube-api-access-5mzkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:58.118103 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.118027 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca" pod "59338868-c02c-4f78-bbc6-e45e09bb6bdb" (UID: "59338868-c02c-4f78-bbc6-e45e09bb6bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:58.217662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.217630 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59338868-c02c-4f78-bbc6-e45e09bb6bdb-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:58.217662 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.217659 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mzkz\" (UniqueName: \"kubernetes.io/projected/59338868-c02c-4f78-bbc6-e45e09bb6bdb-kube-api-access-5mzkz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:51:58.796493 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.796458 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" Oct 31 00:51:58.796756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.796486 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr" event={"ID":"59338868-c02c-4f78-bbc6-e45e09bb6bdb","Type":"ContainerDied","Data":"e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547"} Oct 31 00:51:58.796756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:51:58.796541 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7ca76646d0ba1ac724b1bd67411c57c63e2e667361e2e9324df37c9f8ca3547" Oct 31 00:52:01.966093 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:52:01.966049 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:52:04.186073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:04.186032 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:52:04.186618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:04.186259 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" containerID="cri-o://2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856" gracePeriod=30 Oct 31 00:52:04.705246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:04.705181 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:52:04.705485 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:04.705433 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" containerID="cri-o://a43416f8e50e27bf26c3155ef298a98536686b5972e53f731eda44eb1a0f336c" gracePeriod=30 Oct 31 00:52:05.333264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:05.333228 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:52:05.333680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:05.333456 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" containerID="cri-o://44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88" gracePeriod=30 Oct 31 00:52:09.473877 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.473850 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:52:09.536725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.536695 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.536875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.536803 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.536927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.536907 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.536995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.536976 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537020 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.537122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537073 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537113 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537155 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537070 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.537396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537336 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537347 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.537507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537400 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\") pod \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\" (UID: \"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557\") " Oct 31 00:52:09.537649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537622 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.537782 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.537683 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:09.538359 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.538334 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.538471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.538364 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.538471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.538379 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.538471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.538393 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.538471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.538406 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.540193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.540030 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:09.540193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.540152 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.540462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.540435 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "kube-api-access-vhbm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:09.541440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.541404 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token" pod "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" (UID: "408e0ec0-7e05-4a55-84d9-4ea5d2c7b557"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:09.640927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.640889 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.640927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.640927 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.641128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.640943 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbm5\" (UniqueName: \"kubernetes.io/projected/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-kube-api-access-vhbm5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.641128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.640961 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.918737 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.916633 5011 generic.go:356] "Generic (PLEG): container finished" podID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerID="a43416f8e50e27bf26c3155ef298a98536686b5972e53f731eda44eb1a0f336c" exitCode=0 Oct 31 00:52:09.918737 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.916829 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" event={"ID":"d1b67d25-03a6-40c6-948b-f9d3821a63b8","Type":"ContainerDied","Data":"a43416f8e50e27bf26c3155ef298a98536686b5972e53f731eda44eb1a0f336c"} Oct 31 00:52:09.919825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.919782 5011 generic.go:356] "Generic (PLEG): container finished" podID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerID="2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856" exitCode=0 Oct 31 00:52:09.920762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.920062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" event={"ID":"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557","Type":"ContainerDied","Data":"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856"} Oct 31 00:52:09.920762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.920116 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" event={"ID":"408e0ec0-7e05-4a55-84d9-4ea5d2c7b557","Type":"ContainerDied","Data":"aae52f3185d502f0c20560bd09afb981bd674f1bf7c5559b1f61bcbae25679d9"} Oct 31 00:52:09.920762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.920148 5011 scope.go:122] "RemoveContainer" containerID="2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856" Oct 31 00:52:09.920762 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.920323 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b" Oct 31 00:52:09.947053 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.947026 5011 scope.go:122] "RemoveContainer" containerID="2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856" Oct 31 00:52:09.949246 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:52:09.948698 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856\": container with ID starting with 2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856 not found: ID does not exist" containerID="2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856" Oct 31 00:52:09.949246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.948743 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856"} err="failed to get container status \"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856\": rpc error: code = NotFound desc = could not find container \"2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856\": container with ID starting with 2c30b1825b5c303162de7c2dbfa77a3dad36a9a9861eca3dff2b5c9da3c78856 not found: ID does not exist" Oct 31 00:52:09.984290 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.984263 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:52:09.991847 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.991822 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-rdw5b"] Oct 31 00:52:09.994096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:09.994075 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154042 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154158 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154341 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154514 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154615 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154715 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.155542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.154801 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.161196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.158822 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.161196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.158899 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\") pod \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\" (UID: \"d1b67d25-03a6-40c6-948b-f9d3821a63b8\") " Oct 31 00:52:10.161196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.160606 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.161196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.160885 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.161551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.161201 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:10.162060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.162025 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.162912 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.162864 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.163046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.162897 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.165254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.164109 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.165414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.165386 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "kube-api-access-cftkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.166069 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.166052 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo" pod "d1b67d25-03a6-40c6-948b-f9d3821a63b8" (UID: "d1b67d25-03a6-40c6-948b-f9d3821a63b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260747 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cftkp\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-kube-api-access-cftkp\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260775 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260784 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260792 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260800 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260808 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.260813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260818 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.261105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260826 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.261105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.260835 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b67d25-03a6-40c6-948b-f9d3821a63b8-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.695731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.695703 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:52:10.869698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869664 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.869899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869764 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.869899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869819 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.869899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869885 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869984 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.870061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.869994 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.870145 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.870268 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.870305 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\") pod \"16a9b271-9aa0-43ec-85b9-956294374f3c\" (UID: \"16a9b271-9aa0-43ec-85b9-956294374f3c\") " Oct 31 00:52:10.870700 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.870675 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:10.870918 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.870896 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.871112 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.871092 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.871337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.871315 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.871405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.871344 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a9b271-9aa0-43ec-85b9-956294374f3c-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.871405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.871358 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.871405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.871371 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.872067 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.872041 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.873635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.873588 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "kube-api-access-ltr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.874861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.874836 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:10.874975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.874896 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.876410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.876383 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy" pod "16a9b271-9aa0-43ec-85b9-956294374f3c" (UID: "16a9b271-9aa0-43ec-85b9-956294374f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.928729 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.928705 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" Oct 31 00:52:10.928832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.928719 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt" event={"ID":"d1b67d25-03a6-40c6-948b-f9d3821a63b8","Type":"ContainerDied","Data":"6e29ffe8e64d1729d2f5addb303737735464086eec5c339d13322adfa75c9ebd"} Oct 31 00:52:10.928832 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.928789 5011 scope.go:122] "RemoveContainer" containerID="a43416f8e50e27bf26c3155ef298a98536686b5972e53f731eda44eb1a0f336c" Oct 31 00:52:10.931465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.931439 5011 generic.go:356] "Generic (PLEG): container finished" podID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerID="44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88" exitCode=0 Oct 31 00:52:10.931590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.931470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" event={"ID":"16a9b271-9aa0-43ec-85b9-956294374f3c","Type":"ContainerDied","Data":"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88"} Oct 31 00:52:10.931590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.931490 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" Oct 31 00:52:10.931590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.931506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876" event={"ID":"16a9b271-9aa0-43ec-85b9-956294374f3c","Type":"ContainerDied","Data":"4cb84f6ed77ba320b2904e12f08b5a29b62e76c59bbd301185e22784b5a4eb73"} Oct 31 00:52:10.940176 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.940152 5011 scope.go:122] "RemoveContainer" containerID="44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88" Oct 31 00:52:10.949947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.949929 5011 scope.go:122] "RemoveContainer" containerID="44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88" Oct 31 00:52:10.950184 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:52:10.950166 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88\": container with ID starting with 44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88 not found: ID does not exist" containerID="44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88" Oct 31 00:52:10.950303 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.950277 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88"} err="failed to get container status \"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88\": rpc error: code = NotFound desc = could not find container \"44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88\": container with ID starting with 44e846bc78b5ca2ffddc1c4a289f0bafc58bc33e4fe763cea008142284415a88 not found: ID does not exist" Oct 31 00:52:10.973008 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.972974 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.973008 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.973007 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.973528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.973019 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.973528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.973032 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a9b271-9aa0-43ec-85b9-956294374f3c-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.973528 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.973045 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltr5f\" (UniqueName: \"kubernetes.io/projected/16a9b271-9aa0-43ec-85b9-956294374f3c-kube-api-access-ltr5f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.991279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.991247 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:52:10.996663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:10.996642 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-wzcwt"] Oct 31 00:52:11.030596 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:11.030571 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:52:11.037046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:11.037018 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-ln876"] Oct 31 00:52:11.685217 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:11.685184 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" path="/var/lib/kubelet/pods/16a9b271-9aa0-43ec-85b9-956294374f3c/volumes" Oct 31 00:52:11.685693 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:11.685678 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" path="/var/lib/kubelet/pods/408e0ec0-7e05-4a55-84d9-4ea5d2c7b557/volumes" Oct 31 00:52:11.686145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:11.686130 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" path="/var/lib/kubelet/pods/d1b67d25-03a6-40c6-948b-f9d3821a63b8/volumes" Oct 31 00:52:14.508381 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:14.508333 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq"] Oct 31 00:52:14.517947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:14.517913 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-208e9f32-mg4hq"] Oct 31 00:52:15.692967 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:15.692929 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782ae276-5bf4-4617-b01d-2882f2c36e71" path="/var/lib/kubelet/pods/782ae276-5bf4-4617-b01d-2882f2c36e71/volumes" Oct 31 00:52:26.162233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:26.160793 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:52:31.949675 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:52:31.949638 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:52:46.206435 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:46.206351 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:52:46.206954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:46.206584 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" containerID="cri-o://6e40c587d4cb609d65b2a961eaf05499f550b3876bbac8f70f48c660cd17009a" gracePeriod=30 Oct 31 00:52:48.480200 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480159 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 00:52:48.480922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480899 5011 cpu_manager.go:414] "containerMap: removing container" podUID="59338868-c02c-4f78-bbc6-e45e09bb6bdb" containerName="revoke-keys" Oct 31 00:52:48.480922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480921 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59338868-c02c-4f78-bbc6-e45e09bb6bdb" containerName="revoke-keys" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480936 5011 cpu_manager.go:414] "containerMap: removing container" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480941 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480958 5011 cpu_manager.go:414] "containerMap: removing container" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480965 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480981 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" Oct 31 00:52:48.481079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.480986 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" Oct 31 00:52:48.481431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.481133 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="408e0ec0-7e05-4a55-84d9-4ea5d2c7b557" containerName="istio-proxy" Oct 31 00:52:48.481431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.481148 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b67d25-03a6-40c6-948b-f9d3821a63b8" containerName="istio-proxy" Oct 31 00:52:48.481431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.481168 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="59338868-c02c-4f78-bbc6-e45e09bb6bdb" containerName="revoke-keys" Oct 31 00:52:48.481431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.481178 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a9b271-9aa0-43ec-85b9-956294374f3c" containerName="istio-proxy" Oct 31 00:52:48.485322 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.485297 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.506869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.506828 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 00:52:48.561411 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.561368 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-609843-openshift-default-dockercfg-cprt2\"" Oct 31 00:52:48.592830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.592796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.592978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.592965 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.592996 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593064 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593025 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593140 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593119 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p755d\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593177 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593250 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593323 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.593428 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.593416 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700384 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700416 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700467 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p755d\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700523 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700571 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700683 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700761 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.700963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.700852 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.702249 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.702195 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.702918 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.702886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.703179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.703155 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.703691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.703665 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.704320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.704297 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.721954 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.721929 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.722271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.722249 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.734537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.734466 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p755d\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.745274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.739768 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\") pod \"e2e-inf-b-609843-openshift-default-db7868f45-jmbpz\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:48.842230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:48.816099 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:49.431844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:49.431798 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 00:52:49.433550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:49.433257 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:52:49.433808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:49.433779 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:52:49.433982 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:49.433964 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:52:50.409427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:50.409381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerStarted","Data":"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1"} Oct 31 00:52:50.409888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:50.409458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerStarted","Data":"77d46e9c97bcfff633c53c3edd59ec94f97d724ea4bc2493b5ae387c14c0e4a7"} Oct 31 00:52:50.489837 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:50.489786 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podStartSLOduration=2.489772302 podStartE2EDuration="2.489772302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:50.474726483 +0000 UTC m=+2038.923814124" watchObservedRunningTime="2026-10-31 00:52:50.489772302 +0000 UTC m=+2038.938860002" Oct 31 00:52:50.837240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:50.836120 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:50.845379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:50.845345 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:51.432193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.432158 5011 generic.go:356] "Generic (PLEG): container finished" podID="3ecdd291-ba26-4816-aab6-9a903087606b" containerID="6e40c587d4cb609d65b2a961eaf05499f550b3876bbac8f70f48c660cd17009a" exitCode=0 Oct 31 00:52:51.433128 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.432323 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" event={"ID":"3ecdd291-ba26-4816-aab6-9a903087606b","Type":"ContainerDied","Data":"6e40c587d4cb609d65b2a961eaf05499f550b3876bbac8f70f48c660cd17009a"} Oct 31 00:52:51.433884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.433725 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:51.434831 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.434557 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:52:51.542507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.542480 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:52:51.682730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.682692 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.682915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.682794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.682915 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.682832 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.682946 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683095 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.683055 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683221 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.683184 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.683334 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683506 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.683488 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.683591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.683575 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\") pod \"3ecdd291-ba26-4816-aab6-9a903087606b\" (UID: \"3ecdd291-ba26-4816-aab6-9a903087606b\") " Oct 31 00:52:51.685018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.684992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:51.685242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.685222 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:51.687711 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.687685 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:51.687822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.687758 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:51.687879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.687828 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:51.688327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.688292 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:51.690283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.690155 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:51.692387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.692177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "kube-api-access-lkcww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:51.714450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.714409 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token" pod "3ecdd291-ba26-4816-aab6-9a903087606b" (UID: "3ecdd291-ba26-4816-aab6-9a903087606b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:51.786335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786256 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdd291-ba26-4816-aab6-9a903087606b-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786289 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdd291-ba26-4816-aab6-9a903087606b-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786301 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786315 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcww\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-kube-api-access-lkcww\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786329 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786342 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786353 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786364 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd291-ba26-4816-aab6-9a903087606b-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:51.786657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:51.786374 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdd291-ba26-4816-aab6-9a903087606b-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:52:52.441860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.441811 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" event={"ID":"3ecdd291-ba26-4816-aab6-9a903087606b","Type":"ContainerDied","Data":"c8a0c802fc35e42c3e083a6a86b1bbeec65dfd4fb2c499d17f9c07ed95283e48"} Oct 31 00:52:52.442393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.441889 5011 scope.go:122] "RemoveContainer" containerID="6e40c587d4cb609d65b2a961eaf05499f550b3876bbac8f70f48c660cd17009a" Oct 31 00:52:52.442393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.441961 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd" Oct 31 00:52:52.508012 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.507987 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:52:52.520056 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.520031 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-2b2nd"] Oct 31 00:52:52.780475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:52.780396 5011 scope.go:122] "RemoveContainer" containerID="c3cac4fc29e005931cff475abd37a5b82b1b857b01c447746f228e51c44f7fbc" Oct 31 00:52:53.034906 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:53.034823 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc"] Oct 31 00:52:53.040684 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:53.040651 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a5ebf6-6q4xc"] Oct 31 00:52:53.684648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:53.684611 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" path="/var/lib/kubelet/pods/3ecdd291-ba26-4816-aab6-9a903087606b/volumes" Oct 31 00:52:53.685116 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:52:53.685099 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a696096-a2b1-4168-8f3d-f92c9e82831c" path="/var/lib/kubelet/pods/5a696096-a2b1-4168-8f3d-f92c9e82831c/volumes" Oct 31 00:53:00.135094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.135057 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc"] Oct 31 00:53:00.136066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.136042 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" Oct 31 00:53:00.136149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.136067 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" Oct 31 00:53:00.136286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.136272 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecdd291-ba26-4816-aab6-9a903087606b" containerName="istio-proxy" Oct 31 00:53:00.142320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.142295 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.156243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.156218 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc"] Oct 31 00:53:00.163261 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.163236 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:00.163373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.163362 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:53:00.225481 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.225445 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.225664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.225574 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.225883 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.225858 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjql\" (UniqueName: \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.327299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.327261 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjql\" (UniqueName: \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.327465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.327354 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.327465 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.327421 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.328059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.328038 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.330163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.330132 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.385818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.385754 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjql\" (UniqueName: \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\") pod \"maas-api-revoke-keys-e2e-api-a-68f412-qtdqc\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.459100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.459068 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:00.773015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:00.772984 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc"] Oct 31 00:53:00.773144 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:53:00.773101 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf89086_b599_4728_a104_fc2bc73aba64.slice/crio-cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb WatchSource:0}: Error finding container cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb: Status 404 returned error can't find the container with id cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb Oct 31 00:53:01.549406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:01.549368 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" event={"ID":"bbf89086-b599-4728-a104-fc2bc73aba64","Type":"ContainerStarted","Data":"6aeb29c084e9cb3426dad74bb91a383dab689bd39fac418fbd3b1c046715f891"} Oct 31 00:53:01.549846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:01.549432 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" event={"ID":"bbf89086-b599-4728-a104-fc2bc73aba64","Type":"ContainerStarted","Data":"cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb"} Oct 31 00:53:01.593356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:01.593308 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" podStartSLOduration=1.593292084 podStartE2EDuration="1.593292084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:01.589940552 +0000 UTC m=+2050.039028190" watchObservedRunningTime="2026-10-31 00:53:01.593292084 +0000 UTC m=+2050.042379724" Oct 31 00:53:01.954310 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:53:01.954273 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:53:02.558650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:02.558610 5011 generic.go:356] "Generic (PLEG): container finished" podID="bbf89086-b599-4728-a104-fc2bc73aba64" containerID="6aeb29c084e9cb3426dad74bb91a383dab689bd39fac418fbd3b1c046715f891" exitCode=0 Oct 31 00:53:02.559046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:02.558699 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" event={"ID":"bbf89086-b599-4728-a104-fc2bc73aba64","Type":"ContainerDied","Data":"6aeb29c084e9cb3426dad74bb91a383dab689bd39fac418fbd3b1c046715f891"} Oct 31 00:53:03.761437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.761409 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:03.888629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.888554 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\" (UniqueName: \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\") pod \"bbf89086-b599-4728-a104-fc2bc73aba64\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " Oct 31 00:53:03.888770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.888636 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\") pod \"bbf89086-b599-4728-a104-fc2bc73aba64\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " Oct 31 00:53:03.888770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.888713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\") pod \"bbf89086-b599-4728-a104-fc2bc73aba64\" (UID: \"bbf89086-b599-4728-a104-fc2bc73aba64\") " Oct 31 00:53:03.889058 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.889024 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca" pod "bbf89086-b599-4728-a104-fc2bc73aba64" (UID: "bbf89086-b599-4728-a104-fc2bc73aba64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:03.889354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.889335 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:03.890825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.890800 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca" pod "bbf89086-b599-4728-a104-fc2bc73aba64" (UID: "bbf89086-b599-4728-a104-fc2bc73aba64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:03.890924 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.890804 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql" pod "bbf89086-b599-4728-a104-fc2bc73aba64" (UID: "bbf89086-b599-4728-a104-fc2bc73aba64"). InnerVolumeSpecName "kube-api-access-msjql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:03.990337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.990304 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjql\" (UniqueName: \"kubernetes.io/projected/bbf89086-b599-4728-a104-fc2bc73aba64-kube-api-access-msjql\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:03.990337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:03.990332 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf89086-b599-4728-a104-fc2bc73aba64-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:04.577937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:04.577910 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" Oct 31 00:53:04.577937 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:04.577920 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc" event={"ID":"bbf89086-b599-4728-a104-fc2bc73aba64","Type":"ContainerDied","Data":"cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb"} Oct 31 00:53:04.578143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:04.577986 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd01d717ed2ddca4944970d96a99f0f95fbe160b7907dfad7a26790d9a69f9eb" Oct 31 00:53:07.920316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:07.920279 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:53:07.920800 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:07.920520 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" podUID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" containerName="maas-api" containerID="cri-o://811573f81bbee21ef524d424d76a71516cda461c1c2a0860e85af3e87d818b30" gracePeriod=30 Oct 31 00:53:08.661983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.661905 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" containerID="811573f81bbee21ef524d424d76a71516cda461c1c2a0860e85af3e87d818b30" exitCode=0 Oct 31 00:53:08.665384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.662784 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" event={"ID":"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15","Type":"ContainerDied","Data":"811573f81bbee21ef524d424d76a71516cda461c1c2a0860e85af3e87d818b30"} Oct 31 00:53:08.665384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.662871 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" event={"ID":"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15","Type":"ContainerDied","Data":"bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b"} Oct 31 00:53:08.665384 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.662898 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bacf1d987d34e1a61c30b9a29fb35010f4148e176a6fae94390c42519083891b" Oct 31 00:53:08.671399 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.671374 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:53:08.735868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.735772 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\") pod \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " Oct 31 00:53:08.737556 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.737488 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\") pod \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " Oct 31 00:53:08.738191 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.738171 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\" (UniqueName: \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\") pod \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " Oct 31 00:53:08.738807 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.738786 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca" pod "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" (UID: "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:08.739020 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.738956 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") pod \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\" (UID: \"0b80780e-6fb8-4f48-bd07-e1c8f4f73c15\") " Oct 31 00:53:08.766894 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.766829 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs" pod "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" (UID: "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:08.769666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.769644 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:08.769899 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.769884 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:08.788238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.782774 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls" pod "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" (UID: "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:08.801234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.794430 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8" pod "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" (UID: "0b80780e-6fb8-4f48-bd07-e1c8f4f73c15"). InnerVolumeSpecName "kube-api-access-s82v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:08.874427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.873461 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82v8\" (UniqueName: \"kubernetes.io/projected/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-kube-api-access-s82v8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:08.874427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:08.873504 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:09.111852 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.111822 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp"] Oct 31 00:53:09.117580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.117544 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b1945-pqbmp"] Oct 31 00:53:09.673201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.673171 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd" Oct 31 00:53:09.683597 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.683569 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a" path="/var/lib/kubelet/pods/fd47b3ef-f9b7-432a-ad76-1a1a42b0bc5a/volumes" Oct 31 00:53:09.739828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.739800 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:53:09.751671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:09.751644 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b1945-5bc4b56c6b-rphpd"] Oct 31 00:53:11.685411 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:11.685370 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" path="/var/lib/kubelet/pods/0b80780e-6fb8-4f48-bd07-e1c8f4f73c15/volumes" Oct 31 00:53:13.704744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:13.704713 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr"] Oct 31 00:53:13.715952 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:13.715922 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d21c7-kddcr"] Oct 31 00:53:15.684300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:15.684268 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59338868-c02c-4f78-bbc6-e45e09bb6bdb" path="/var/lib/kubelet/pods/59338868-c02c-4f78-bbc6-e45e09bb6bdb/volumes" Oct 31 00:53:17.415029 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.414995 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:53:17.415647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415632 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" containerName="maas-api" Oct 31 00:53:17.415728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415648 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" containerName="maas-api" Oct 31 00:53:17.415728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415683 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bbf89086-b599-4728-a104-fc2bc73aba64" containerName="revoke-keys" Oct 31 00:53:17.415728 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415688 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf89086-b599-4728-a104-fc2bc73aba64" containerName="revoke-keys" Oct 31 00:53:17.415863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415782 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80780e-6fb8-4f48-bd07-e1c8f4f73c15" containerName="maas-api" Oct 31 00:53:17.415863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.415797 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf89086-b599-4728-a104-fc2bc73aba64" containerName="revoke-keys" Oct 31 00:53:17.419078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.419052 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.438620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.438590 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-82d887-openshift-default-dockercfg-zdn5c\"" Oct 31 00:53:17.467284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.467253 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:53:17.536196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536142 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536234 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536298 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536358 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536406 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536524 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536568 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnp2\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536741 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.536814 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.536765 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.637887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.637848 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.637942 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.637970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnp2\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638173 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638199 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638350 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638333 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638424 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638402 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638436 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638477 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638693 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638727 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638696 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.638945 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.638906 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.640841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.640818 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.641080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.641055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.662423 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.662398 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.663900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.663874 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnp2\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:17.736534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:17.736454 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:18.085305 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:53:18.085278 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4694b61_13ca_470c_8139_0d07c715f633.slice/crio-09d36ccaf69f050a079f1f8c141a71d52fe527aee5f21d6c0c78b888eba9c51b WatchSource:0}: Error finding container 09d36ccaf69f050a079f1f8c141a71d52fe527aee5f21d6c0c78b888eba9c51b: Status 404 returned error can't find the container with id 09d36ccaf69f050a079f1f8c141a71d52fe527aee5f21d6c0c78b888eba9c51b Oct 31 00:53:18.086440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.086416 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:53:18.087968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.087937 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:53:18.088035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.088001 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:53:18.088035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.088029 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:53:18.792192 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.792147 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" event={"ID":"c4694b61-13ca-470c-8139-0d07c715f633","Type":"ContainerStarted","Data":"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa"} Oct 31 00:53:18.794293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.794263 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" event={"ID":"c4694b61-13ca-470c-8139-0d07c715f633","Type":"ContainerStarted","Data":"09d36ccaf69f050a079f1f8c141a71d52fe527aee5f21d6c0c78b888eba9c51b"} Oct 31 00:53:18.868292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:18.868224 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podStartSLOduration=1.8681902959999999 podStartE2EDuration="1.868190296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:18.861958379 +0000 UTC m=+2067.311046018" watchObservedRunningTime="2026-10-31 00:53:18.868190296 +0000 UTC m=+2067.317277936" Oct 31 00:53:19.759408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:19.755645 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:19.759408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:19.758316 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:19.759408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:19.758371 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:20.741229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:20.739561 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:20.741229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:20.739618 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:21.231233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.217710 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 00:53:21.270226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.263510 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 00:53:21.270226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.263607 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.279256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.278619 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c6d97d-serving-cert\"" Oct 31 00:53:21.367452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.367406 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.367644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.367465 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.367644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.367575 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgwb\" (UniqueName: \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.367644 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.367642 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.469721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.469679 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.469934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.469743 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.469934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.469848 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgwb\" (UniqueName: \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.469934 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.469924 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.470623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.470589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.472611 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.472580 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.473031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.473011 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.496713 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.496642 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgwb\" (UniqueName: \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\") pod \"maas-api-e2e-inf-b-c6d97d-544557c654-k95wp\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.591236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.586517 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:21.803345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.801758 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:21.803345 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.801821 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:21.830234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.817112 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:53:21.830234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:21.817366 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" containerID="cri-o://9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3" gracePeriod=30 Oct 31 00:53:22.462769 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:22.462733 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 00:53:22.465856 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:53:22.465796 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda76ea30_ff43_4aed_ac48_5483bc01983c.slice/crio-92f131ab4cff3eeeb6f54ebd7e42cb8942dee36443354ac7133782458ec2b982 WatchSource:0}: Error finding container 92f131ab4cff3eeeb6f54ebd7e42cb8942dee36443354ac7133782458ec2b982: Status 404 returned error can't find the container with id 92f131ab4cff3eeeb6f54ebd7e42cb8942dee36443354ac7133782458ec2b982 Oct 31 00:53:22.468097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:22.468080 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:53:22.740797 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:22.740709 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:22.741077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:22.741042 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:23.037379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:23.037288 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" event={"ID":"da76ea30-ff43-4aed-ac48-5483bc01983c","Type":"ContainerStarted","Data":"92f131ab4cff3eeeb6f54ebd7e42cb8942dee36443354ac7133782458ec2b982"} Oct 31 00:53:23.750233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:23.749248 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:23.750233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:23.749308 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:24.085245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.072073 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:53:24.100152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.100120 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.108374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.108332 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" event={"ID":"da76ea30-ff43-4aed-ac48-5483bc01983c","Type":"ContainerStarted","Data":"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab"} Oct 31 00:53:24.108544 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.108412 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:24.145230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.120671 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:53:24.187132 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.183856 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bdb61f-serving-cert\"" Oct 31 00:53:24.187351 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.186926 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hk25\" (UniqueName: \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.187496 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.187478 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.187654 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.187639 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.189532 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.189507 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.291917 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.291669 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.294681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.293919 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hk25\" (UniqueName: \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.294681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.294652 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.294681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.294677 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.296305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.296277 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.297734 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.297717 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.297855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.297831 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.366274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.349265 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hk25\" (UniqueName: \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\") pod \"maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.507499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.507463 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:24.740577 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.740540 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:53:24.740846 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.740818 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 00:53:24.948134 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.948021 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" podStartSLOduration=3.307409576 podStartE2EDuration="3.948000532s" totalImagesPullingTime="640.590956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:24.348560107 +0000 UTC m=+2072.797647748" watchObservedRunningTime="2026-10-31 00:53:24.948000532 +0000 UTC m=+2073.397088172" Oct 31 00:53:24.948980 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:24.948949 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:53:25.143230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:25.142748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" event={"ID":"0ed63ec6-7272-4463-b538-9c076a15155b","Type":"ContainerStarted","Data":"e3e31dd3fee015b2f3d13283b85cffb0a2161ff81422f684f8937ec9b92a1b88"} Oct 31 00:53:25.782893 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:25.782858 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:25.783434 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:25.783400 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:25.787265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:25.787237 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:53:26.184784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:26.184748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" event={"ID":"0ed63ec6-7272-4463-b538-9c076a15155b","Type":"ContainerStarted","Data":"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2"} Oct 31 00:53:26.185277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:26.184890 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:26.247430 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:26.247369 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" podStartSLOduration=1.912255149 podStartE2EDuration="2.247353684s" totalImagesPullingTime="335.098535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:26.242619962 +0000 UTC m=+2074.691707602" watchObservedRunningTime="2026-10-31 00:53:26.247353684 +0000 UTC m=+2074.696441326" Oct 31 00:53:27.474822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.474796 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:53:27.495094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495067 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495142 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495167 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495263 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495345 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495372 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495409 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495487 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495492 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:27.495618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495517 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\") pod \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\" (UID: \"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78\") " Oct 31 00:53:27.495618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495547 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:27.495618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495581 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:27.495891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495616 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:27.495891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.495725 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:27.496387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.496362 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.496488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.496391 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.496488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.496410 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.496488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.496424 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.496488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.496437 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.497695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.497668 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:27.497781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.497705 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:27.497781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.497719 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:27.497854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.497794 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc" pod "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" (UID: "e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78"). InnerVolumeSpecName "kube-api-access-rsnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:27.598009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.597979 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.598009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.598007 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.598009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.598018 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:27.598292 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:27.598027 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnnc\" (UniqueName: \"kubernetes.io/projected/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78-kube-api-access-rsnnc\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:28.202032 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.201996 5011 generic.go:356] "Generic (PLEG): container finished" podID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerID="9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3" exitCode=0 Oct 31 00:53:28.202257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.202056 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" Oct 31 00:53:28.202257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.202078 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerDied","Data":"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3"} Oct 31 00:53:28.202257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.202135 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4" event={"ID":"e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78","Type":"ContainerDied","Data":"7f2569fa96f394d3b853aaa74c3c33633590f83208a5d1f69773c13c7b63dd95"} Oct 31 00:53:28.202257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.202161 5011 scope.go:122] "RemoveContainer" containerID="9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3" Oct 31 00:53:28.213537 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.213519 5011 scope.go:122] "RemoveContainer" containerID="eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4" Oct 31 00:53:28.230828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.230809 5011 scope.go:122] "RemoveContainer" containerID="9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3" Oct 31 00:53:28.231134 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:53:28.231113 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3\": container with ID starting with 9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3 not found: ID does not exist" containerID="9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3" Oct 31 00:53:28.231195 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.231144 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3"} err="failed to get container status \"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3\": rpc error: code = NotFound desc = could not find container \"9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3\": container with ID starting with 9b48d23b8d682528f8863d226f6f99182d85b19c7557e3a9323b05dd6f5d9fa3 not found: ID does not exist" Oct 31 00:53:28.231195 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.231161 5011 scope.go:122] "RemoveContainer" containerID="eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4" Oct 31 00:53:28.231450 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:53:28.231430 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4\": container with ID starting with eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4 not found: ID does not exist" containerID="eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4" Oct 31 00:53:28.231512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.231463 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4"} err="failed to get container status \"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4\": rpc error: code = NotFound desc = could not find container \"eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4\": container with ID starting with eec33290af3da391ab9c1f0322b71cf5a8baf94b2dd6feed4d2bd7f23cc9d9b4 not found: ID does not exist" Oct 31 00:53:28.280785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.280750 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:53:28.292682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:28.292658 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3d21c7-openshift-default-5dbdb59c99-gz7q4"] Oct 31 00:53:29.684862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:29.684821 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" path="/var/lib/kubelet/pods/e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78/volumes" Oct 31 00:53:30.150094 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:30.150055 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 00:53:31.948535 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:53:31.948493 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:53:32.201173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:32.201093 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:53:33.729196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.729164 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 00:53:33.730109 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.730087 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:53:33.730153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.730113 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:53:33.730362 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.730348 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:53:33.730417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.730368 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:53:33.736195 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.736174 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.755144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.755119 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-609843-serving-cert\"" Oct 31 00:53:33.757840 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.757810 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 00:53:33.837981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.837947 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.837981 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.837985 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.838242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.838123 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.838242 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.838188 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwmz\" (UniqueName: \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.941405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.940647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.941405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.940704 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.941405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.940785 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.941405 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.940908 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwmz\" (UniqueName: \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.948185 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.941752 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.956467 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.956428 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.984254 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.984164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:33.992681 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:33.992609 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwmz\" (UniqueName: \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\") pod \"maas-api-e2e-inf-b-609843-599df96497-5drnw\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:34.063563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:34.063517 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:34.391889 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:53:34.391855 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55466fa_794d_4f46_87a0_78ed08b1ec89.slice/crio-15ef8ca5b7bbd8a37146f52ae95d5fec3a01fc02a769e0f2d040ae866813a8d6 WatchSource:0}: Error finding container 15ef8ca5b7bbd8a37146f52ae95d5fec3a01fc02a769e0f2d040ae866813a8d6: Status 404 returned error can't find the container with id 15ef8ca5b7bbd8a37146f52ae95d5fec3a01fc02a769e0f2d040ae866813a8d6 Oct 31 00:53:34.392241 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:34.392183 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 00:53:35.272464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:35.272425 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" event={"ID":"e55466fa-794d-4f46-87a0-78ed08b1ec89","Type":"ContainerStarted","Data":"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15"} Oct 31 00:53:35.272938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:35.272489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" event={"ID":"e55466fa-794d-4f46-87a0-78ed08b1ec89","Type":"ContainerStarted","Data":"15ef8ca5b7bbd8a37146f52ae95d5fec3a01fc02a769e0f2d040ae866813a8d6"} Oct 31 00:53:35.272938 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:35.272516 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:35.328232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:35.327740 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" podStartSLOduration=1.93209495 podStartE2EDuration="2.327715395s" totalImagesPullingTime="395.620445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:35.315687059 +0000 UTC m=+2083.764774699" watchObservedRunningTime="2026-10-31 00:53:35.327715395 +0000 UTC m=+2083.776803048" Oct 31 00:53:41.285354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:41.285324 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 00:53:50.753233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:50.753079 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:53:52.244377 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:52.244345 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:53:52.254592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:52.254560 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:53:52.858984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:52.858943 5011 scope.go:122] "RemoveContainer" containerID="08e5322f1fbb787971d77fc46bb3cd4b79dba1bb003426786fe62ba584c85cb4" Oct 31 00:53:52.911400 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:52.911371 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:53:53.603231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:53.596697 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" containerID="cri-o://99055bd1bccf219af7b18a7292f08298b227385fb0ad29c277d4a5340a601fa4" gracePeriod=30 Oct 31 00:53:54.139231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.118270 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 00:53:54.139231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.119328 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.184236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.168721 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.203229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.190285 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.397194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.397070 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 31 00:53:54.618518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.618476 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 00:53:54.633260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.630633 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.703669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.703570 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.867620 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.867508 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.900851 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:54.899285 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:58.770647 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.770605 5011 generic.go:356] "Generic (PLEG): container finished" podID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerID="99055bd1bccf219af7b18a7292f08298b227385fb0ad29c277d4a5340a601fa4" exitCode=0 Oct 31 00:53:58.771023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.770678 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" event={"ID":"8652ffe7-53fd-4b05-a202-9b51be480c88","Type":"ContainerDied","Data":"99055bd1bccf219af7b18a7292f08298b227385fb0ad29c277d4a5340a601fa4"} Oct 31 00:53:58.941328 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.940806 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:53:58.969088 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969055 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969289 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969269 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969354 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969329 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969411 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969499 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969484 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969521 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.969565 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969553 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969697 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969770 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\") pod \"8652ffe7-53fd-4b05-a202-9b51be480c88\" (UID: \"8652ffe7-53fd-4b05-a202-9b51be480c88\") " Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.969766 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970073 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970195 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:58.970299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970263 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:58.970717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970602 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8652ffe7-53fd-4b05-a202-9b51be480c88-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:58.970717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970627 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:58.970717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970642 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:58.970717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970656 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:58.970717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.970670 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:58.973060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.973031 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:58.973173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.973060 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "kube-api-access-xqd2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:58.973504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.973481 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:58.973770 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:58.973749 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo" pod "8652ffe7-53fd-4b05-a202-9b51be480c88" (UID: "8652ffe7-53fd-4b05-a202-9b51be480c88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:59.072869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.072830 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqd2f\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-kube-api-access-xqd2f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:59.072869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.072870 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:59.072869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.072886 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:59.073201 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.072900 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8652ffe7-53fd-4b05-a202-9b51be480c88-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:53:59.781073 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.781032 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" event={"ID":"8652ffe7-53fd-4b05-a202-9b51be480c88","Type":"ContainerDied","Data":"7d8b8bedcd68091048d1ca3ea56ea8bf8d58572b0b2f185ff777b0e8ae2c8192"} Oct 31 00:53:59.781575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.781112 5011 scope.go:122] "RemoveContainer" containerID="99055bd1bccf219af7b18a7292f08298b227385fb0ad29c277d4a5340a601fa4" Oct 31 00:53:59.781575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.781307 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz" Oct 31 00:53:59.854873 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.854829 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:53:59.867253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:53:59.866779 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-b22gz"] Oct 31 00:54:00.267141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.267095 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 00:54:00.268501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.268469 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:54:00.268501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.268496 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f9a3b5-9dc3-4082-b9c1-5b0f2c249a78" containerName="istio-proxy" Oct 31 00:54:00.268716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.268548 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" Oct 31 00:54:00.268716 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.268556 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" Oct 31 00:54:00.268932 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.268912 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" containerName="istio-proxy" Oct 31 00:54:00.282727 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.282686 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 00:54:00.282879 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.282820 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.298725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.298684 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b87c62-serving-cert\"" Oct 31 00:54:00.392551 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.392505 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.392784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.392757 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pk4n\" (UniqueName: \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.392830 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.392818 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.393035 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.393004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.495033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.494993 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk4n\" (UniqueName: \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.495278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.495072 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.495278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.495188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.495426 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.495278 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.496188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.496156 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.497856 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.497827 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.498196 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.498174 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.523904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.523828 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pk4n\" (UniqueName: \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\") pod \"maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.615098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.615065 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:00.933102 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:00.933068 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 00:54:00.934005 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:54:00.933980 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72be45f8_51e7_42e5_8097_25d7b6f56a33.slice/crio-8c7677122d21ae506ba54b335361c918382f5700b47bc6be442bb8f738078182 WatchSource:0}: Error finding container 8c7677122d21ae506ba54b335361c918382f5700b47bc6be442bb8f738078182: Status 404 returned error can't find the container with id 8c7677122d21ae506ba54b335361c918382f5700b47bc6be442bb8f738078182 Oct 31 00:54:01.684723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:01.684688 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8652ffe7-53fd-4b05-a202-9b51be480c88" path="/var/lib/kubelet/pods/8652ffe7-53fd-4b05-a202-9b51be480c88/volumes" Oct 31 00:54:01.800406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:01.800365 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" event={"ID":"72be45f8-51e7-42e5-8097-25d7b6f56a33","Type":"ContainerStarted","Data":"87792358fdadb08116b4e7bea6f8681384186478a8000d1923d46b7145ca2e92"} Oct 31 00:54:01.800576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:01.800434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:01.800576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:01.800460 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" event={"ID":"72be45f8-51e7-42e5-8097-25d7b6f56a33","Type":"ContainerStarted","Data":"8c7677122d21ae506ba54b335361c918382f5700b47bc6be442bb8f738078182"} Oct 31 00:54:01.950130 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:54:01.949431 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:54:02.509880 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.509824 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" podStartSLOduration=2.231793071 podStartE2EDuration="2.509808819s" totalImagesPullingTime="278.015748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:01.860603827 +0000 UTC m=+2110.309691466" watchObservedRunningTime="2026-10-31 00:54:02.509808819 +0000 UTC m=+2110.958896458" Oct 31 00:54:02.512622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.512597 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:54:02.517231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.517199 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.542449 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.542426 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:54:02.662721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.662690 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2w9\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.662990 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663032 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663106 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663177 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663269 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663299 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663386 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.664888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.663423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.764942 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.764875 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.764942 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.764924 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.764955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.764997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765035 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765119 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765074 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765280 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765359 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2w9\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765510 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765537 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.765818 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.765588 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.766148 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.766127 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.766188 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.766163 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.767461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.767441 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.767634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.767618 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.798439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.798416 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.799479 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.799458 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2w9\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:02.809702 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.809680 5011 generic.go:356] "Generic (PLEG): container finished" podID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerID="46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1" exitCode=0 Oct 31 00:54:02.809783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.809748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerDied","Data":"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1"} Oct 31 00:54:02.810456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.810441 5011 scope.go:122] "RemoveContainer" containerID="46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1" Oct 31 00:54:02.812000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.811975 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:02.812061 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.812041 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:02.812100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.812072 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:02.833808 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:02.833785 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:03.195083 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:54:03.195054 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3430a600_4381_4f5f_b969_579dbad40cdf.slice/crio-8af035ffd6750bf2f9664cd4922e20043828241a1834e10dc57b1a2f65f13439 WatchSource:0}: Error finding container 8af035ffd6750bf2f9664cd4922e20043828241a1834e10dc57b1a2f65f13439: Status 404 returned error can't find the container with id 8af035ffd6750bf2f9664cd4922e20043828241a1834e10dc57b1a2f65f13439 Oct 31 00:54:03.197621 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.197593 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:54:03.198366 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.198335 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:03.198452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.198401 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:03.198452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.198428 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:54:03.832669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.832620 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" event={"ID":"3430a600-4381-4f5f-b969-579dbad40cdf","Type":"ContainerStarted","Data":"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915"} Oct 31 00:54:03.832896 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.832695 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" event={"ID":"3430a600-4381-4f5f-b969-579dbad40cdf","Type":"ContainerStarted","Data":"8af035ffd6750bf2f9664cd4922e20043828241a1834e10dc57b1a2f65f13439"} Oct 31 00:54:03.836462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.836428 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerStarted","Data":"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097"} Oct 31 00:54:03.838670 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.838491 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:54:03.838795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.838760 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:03.840309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.840268 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:03.840512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.840479 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:03.900534 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:03.900477 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podStartSLOduration=1.900462835 podStartE2EDuration="1.900462835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:03.897989832 +0000 UTC m=+2112.347077471" watchObservedRunningTime="2026-10-31 00:54:03.900462835 +0000 UTC m=+2112.349550474" Oct 31 00:54:04.843974 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:04.843554 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:04.843974 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:04.843616 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:04.854843 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:04.854813 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:04.855087 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:04.855056 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:05.837231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:05.834618 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:05.837231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:05.834684 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:05.837231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:05.835390 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:05.837231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:05.835435 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:06.444404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:06.444362 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:54:06.834732 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:06.834700 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:06.835673 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:06.835645 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:06.835976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:06.835952 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:06.836173 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:06.835919 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:07.100898 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.100797 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:54:07.106957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.106920 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 00:54:07.116319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.116267 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:54:07.162407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.162363 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:54:07.702589 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.702547 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc"] Oct 31 00:54:07.712702 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.712677 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-68f412-qtdqc"] Oct 31 00:54:07.816629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.816597 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 00:54:07.848232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.847316 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:07.848232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.847378 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:07.848232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.847714 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:07.848232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:07.847742 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:08.836824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:08.836784 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:08.837457 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:08.837425 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:08.837956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:08.837932 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:08.838097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:08.838079 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:09.690709 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:09.690672 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf89086-b599-4728-a104-fc2bc73aba64" path="/var/lib/kubelet/pods/bbf89086-b599-4728-a104-fc2bc73aba64/volumes" Oct 31 00:54:09.841230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:09.840990 5011 patch_prober.go:29] interesting pod/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:54:09.841230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:09.841068 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 00:54:09.841891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:09.841286 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:09.841891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:09.841348 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:10.838231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:10.837790 5011 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:54:10.838231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:10.837875 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 00:54:10.859284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:10.856692 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:54:10.859284 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:10.858152 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 00:54:11.368991 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.368957 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l"] Oct 31 00:54:11.376023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.375996 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.389930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.389902 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l"] Oct 31 00:54:11.397452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.397425 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:11.397702 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.397686 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:54:11.438407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.438360 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.438795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.438773 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.439026 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.439007 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfks\" (UniqueName: \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.540557 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.540521 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.540836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.540700 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.540836 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.540802 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfks\" (UniqueName: \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.541472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.541442 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.545616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.545589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.565015 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.564995 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfks\" (UniqueName: \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\") pod \"maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.727515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.727437 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:11.858227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.856515 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:11.871971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.860610 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:11.871971 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:11.862336 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:54:12.198629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:12.198596 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l"] Oct 31 00:54:12.308005 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:12.307965 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" event={"ID":"45119cea-00e4-4a72-8dbe-0261215d0f34","Type":"ContainerStarted","Data":"c636237bae043f03971b7ad38e8db2df4f41d28440acdcd6e4f8c94329b1c11f"} Oct 31 00:54:13.317542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:13.317481 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" event={"ID":"45119cea-00e4-4a72-8dbe-0261215d0f34","Type":"ContainerStarted","Data":"245b868802f3135bdca61a0126fd9b49178a8ec305e6880da66571d657a5cc21"} Oct 31 00:54:13.360779 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:13.360723 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" podStartSLOduration=2.360708677 podStartE2EDuration="2.360708677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:13.358471433 +0000 UTC m=+2121.807559075" watchObservedRunningTime="2026-10-31 00:54:13.360708677 +0000 UTC m=+2121.809796394" Oct 31 00:54:14.328678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:14.328639 5011 generic.go:356] "Generic (PLEG): container finished" podID="45119cea-00e4-4a72-8dbe-0261215d0f34" containerID="245b868802f3135bdca61a0126fd9b49178a8ec305e6880da66571d657a5cc21" exitCode=0 Oct 31 00:54:14.329138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:14.328683 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" event={"ID":"45119cea-00e4-4a72-8dbe-0261215d0f34","Type":"ContainerDied","Data":"245b868802f3135bdca61a0126fd9b49178a8ec305e6880da66571d657a5cc21"} Oct 31 00:54:15.509780 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.509756 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:15.557050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.557010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\" (UniqueName: \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\") pod \"45119cea-00e4-4a72-8dbe-0261215d0f34\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " Oct 31 00:54:15.557821 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.557797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\") pod \"45119cea-00e4-4a72-8dbe-0261215d0f34\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " Oct 31 00:54:15.558470 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.558340 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca" pod "45119cea-00e4-4a72-8dbe-0261215d0f34" (UID: "45119cea-00e4-4a72-8dbe-0261215d0f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:15.558992 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.558973 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\") pod \"45119cea-00e4-4a72-8dbe-0261215d0f34\" (UID: \"45119cea-00e4-4a72-8dbe-0261215d0f34\") " Oct 31 00:54:15.560168 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.560142 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks" pod "45119cea-00e4-4a72-8dbe-0261215d0f34" (UID: "45119cea-00e4-4a72-8dbe-0261215d0f34"). InnerVolumeSpecName "kube-api-access-vbfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:15.560583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.560565 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfks\" (UniqueName: \"kubernetes.io/projected/45119cea-00e4-4a72-8dbe-0261215d0f34-kube-api-access-vbfks\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:15.560736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.560722 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:15.562312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.561784 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca" pod "45119cea-00e4-4a72-8dbe-0261215d0f34" (UID: "45119cea-00e4-4a72-8dbe-0261215d0f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:15.585893 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.585834 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs"] Oct 31 00:54:15.587097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.587071 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45119cea-00e4-4a72-8dbe-0261215d0f34" containerName="revoke-keys" Oct 31 00:54:15.587279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.587099 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45119cea-00e4-4a72-8dbe-0261215d0f34" containerName="revoke-keys" Oct 31 00:54:15.587395 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.587292 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45119cea-00e4-4a72-8dbe-0261215d0f34" containerName="revoke-keys" Oct 31 00:54:15.591944 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.591815 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.607741 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.606177 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs"] Oct 31 00:54:15.664256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.664199 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.664447 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.664331 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gd6l\" (UniqueName: \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.664447 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.664421 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.666775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.664738 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45119cea-00e4-4a72-8dbe-0261215d0f34-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:15.765606 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.765565 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gd6l\" (UniqueName: \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.765777 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.765637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.765841 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.765827 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.766484 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.766459 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.768033 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.768006 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.789157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.789128 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gd6l\" (UniqueName: \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\") pod \"maas-api-revoke-keys-e2e-api-b-82d887-5rlgs\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:15.924458 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:15.924384 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:16.237291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:16.237262 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs"] Oct 31 00:54:16.237828 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:54:16.237802 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eab3186_6f90_4399_b6d1_21c54c57fea6.slice/crio-36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a WatchSource:0}: Error finding container 36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a: Status 404 returned error can't find the container with id 36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a Oct 31 00:54:16.351461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:16.351431 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" Oct 31 00:54:16.351461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:16.351449 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l" event={"ID":"45119cea-00e4-4a72-8dbe-0261215d0f34","Type":"ContainerDied","Data":"c636237bae043f03971b7ad38e8db2df4f41d28440acdcd6e4f8c94329b1c11f"} Oct 31 00:54:16.351667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:16.351503 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c636237bae043f03971b7ad38e8db2df4f41d28440acdcd6e4f8c94329b1c11f" Oct 31 00:54:16.353265 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:16.353238 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" event={"ID":"8eab3186-6f90-4399-b6d1-21c54c57fea6","Type":"ContainerStarted","Data":"36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a"} Oct 31 00:54:17.362731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:17.362694 5011 generic.go:356] "Generic (PLEG): container finished" podID="8eab3186-6f90-4399-b6d1-21c54c57fea6" containerID="9d0d7edd40ee01cf8afed8646241344eb4a96863c5e174bfe2b13db76c0b7831" exitCode=0 Oct 31 00:54:17.363133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:17.362739 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" event={"ID":"8eab3186-6f90-4399-b6d1-21c54c57fea6","Type":"ContainerDied","Data":"9d0d7edd40ee01cf8afed8646241344eb4a96863c5e174bfe2b13db76c0b7831"} Oct 31 00:54:18.575930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.575902 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:18.710347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.710277 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\") pod \"8eab3186-6f90-4399-b6d1-21c54c57fea6\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " Oct 31 00:54:18.710488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.710445 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\") pod \"8eab3186-6f90-4399-b6d1-21c54c57fea6\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " Oct 31 00:54:18.710541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.710497 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\" (UniqueName: \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\") pod \"8eab3186-6f90-4399-b6d1-21c54c57fea6\" (UID: \"8eab3186-6f90-4399-b6d1-21c54c57fea6\") " Oct 31 00:54:18.710663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.710637 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca" pod "8eab3186-6f90-4399-b6d1-21c54c57fea6" (UID: "8eab3186-6f90-4399-b6d1-21c54c57fea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:18.711065 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.711040 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:18.712697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.712675 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca" pod "8eab3186-6f90-4399-b6d1-21c54c57fea6" (UID: "8eab3186-6f90-4399-b6d1-21c54c57fea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:18.712781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.712720 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l" pod "8eab3186-6f90-4399-b6d1-21c54c57fea6" (UID: "8eab3186-6f90-4399-b6d1-21c54c57fea6"). InnerVolumeSpecName "kube-api-access-8gd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:18.812550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.812522 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eab3186-6f90-4399-b6d1-21c54c57fea6-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:18.812550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:18.812551 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gd6l\" (UniqueName: \"kubernetes.io/projected/8eab3186-6f90-4399-b6d1-21c54c57fea6-kube-api-access-8gd6l\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:19.379995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:19.379963 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" Oct 31 00:54:19.379995 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:19.379973 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs" event={"ID":"8eab3186-6f90-4399-b6d1-21c54c57fea6","Type":"ContainerDied","Data":"36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a"} Oct 31 00:54:19.380337 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:19.380018 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36dc5005dc4710d780dd6c0b2f698ef6e0737fba0c0389de1cf916282d602e1a" Oct 31 00:54:31.948990 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:54:31.948952 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:54:34.105666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:34.105628 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw"] Oct 31 00:54:34.123685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:34.123650 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbb2511-f4dbw"] Oct 31 00:54:35.684233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:35.684175 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe38a8e-0401-45cd-85d6-14cfce83ae1f" path="/var/lib/kubelet/pods/2fe38a8e-0401-45cd-85d6-14cfce83ae1f/volumes" Oct 31 00:54:40.051538 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:40.051497 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5"] Oct 31 00:54:40.064553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:40.064520 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461c4443-vrfp5"] Oct 31 00:54:41.687374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:41.687340 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10398cf9-a1f3-470b-8dd6-cab5c17e7434" path="/var/lib/kubelet/pods/10398cf9-a1f3-470b-8dd6-cab5c17e7434/volumes" Oct 31 00:54:42.100613 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:42.100573 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:54:49.942936 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.942900 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh"] Oct 31 00:54:49.943904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.943882 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8eab3186-6f90-4399-b6d1-21c54c57fea6" containerName="revoke-keys" Oct 31 00:54:49.943904 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.943904 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eab3186-6f90-4399-b6d1-21c54c57fea6" containerName="revoke-keys" Oct 31 00:54:49.944045 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.944036 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eab3186-6f90-4399-b6d1-21c54c57fea6" containerName="revoke-keys" Oct 31 00:54:49.947972 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.947954 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:49.971617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.971582 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh"] Oct 31 00:54:49.978787 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.978763 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:54:49.978926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:49.978820 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:50.062581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.062539 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql8b\" (UniqueName: \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.062789 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.062748 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.062864 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.062851 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.164595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.164551 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.164824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.164800 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql8b\" (UniqueName: \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.165464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.165441 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.166114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.166085 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.168355 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.168325 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.174950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.174925 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:54:50.175153 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.175127 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" containerID="cri-o://b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd" gracePeriod=30 Oct 31 00:54:50.191410 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.191383 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql8b\" (UniqueName: \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\") pod \"maas-api-revoke-sub-4ddbd505-7rrnh\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.262320 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.262253 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:50.603510 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.602744 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh"] Oct 31 00:54:50.612122 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:54:50.612080 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5b0eb3_20fb_4cf7_b1c9_3f819c804ee8.slice/crio-f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e WatchSource:0}: Error finding container f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e: Status 404 returned error can't find the container with id f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e Oct 31 00:54:50.709133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.709091 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" event={"ID":"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8","Type":"ContainerStarted","Data":"f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e"} Oct 31 00:54:50.965822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.965784 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:54:50.966247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:50.966023 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" containerID="cri-o://4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd" gracePeriod=30 Oct 31 00:54:51.724257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:51.720553 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" event={"ID":"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8","Type":"ContainerStarted","Data":"17fc0e04f8563ca07e8ef54e4079834c99c49ac819b7afb071a912d55ca763dc"} Oct 31 00:54:51.767960 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:51.767907 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" podStartSLOduration=2.767892666 podStartE2EDuration="2.767892666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:51.764398436 +0000 UTC m=+2160.213486096" watchObservedRunningTime="2026-10-31 00:54:51.767892666 +0000 UTC m=+2160.216980304" Oct 31 00:54:52.731588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:52.731548 5011 generic.go:356] "Generic (PLEG): container finished" podID="eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" containerID="17fc0e04f8563ca07e8ef54e4079834c99c49ac819b7afb071a912d55ca763dc" exitCode=0 Oct 31 00:54:52.731979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:52.731622 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" event={"ID":"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8","Type":"ContainerDied","Data":"17fc0e04f8563ca07e8ef54e4079834c99c49ac819b7afb071a912d55ca763dc"} Oct 31 00:54:53.930223 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.930189 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:53.965258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.964714 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\") pod \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " Oct 31 00:54:53.965258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.964934 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\" (UniqueName: \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\") pod \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " Oct 31 00:54:53.965258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.964976 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\") pod \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\" (UID: \"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8\") " Oct 31 00:54:53.966407 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.966377 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca" pod "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" (UID: "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:53.967154 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.967126 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca" pod "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" (UID: "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:53.967659 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:53.967625 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b" pod "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" (UID: "eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8"). InnerVolumeSpecName "kube-api-access-9ql8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:54.067724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.067692 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:54.067724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.067724 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ql8b\" (UniqueName: \"kubernetes.io/projected/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-kube-api-access-9ql8b\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:54.067961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.067735 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:54.751794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.751762 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" Oct 31 00:54:54.751794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.751778 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh" event={"ID":"eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8","Type":"ContainerDied","Data":"f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e"} Oct 31 00:54:54.752059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.751831 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b2ab97c4f4b429865c8692ec0ff76fa52835a89e93d08aa4a8ceb9f81cce8e" Oct 31 00:54:54.936884 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.936847 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:54:54.937293 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:54.936919 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:54:55.497328 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.497301 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:54:55.590667 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.590822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590690 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.590822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590731 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.590822 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.591007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590838 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.591007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.590957 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.591110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.591026 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.591110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.591069 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.591316 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.591295 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\") pod \"0680007d-233b-4386-bb5c-6f6595f2bdab\" (UID: \"0680007d-233b-4386-bb5c-6f6595f2bdab\") " Oct 31 00:54:55.592122 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.592097 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:55.592396 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.592376 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:55.592490 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.592455 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:55.592590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.592573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:55.592757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.592741 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:55.597501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.595117 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:55.597501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.595250 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:55.597501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.595345 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:55.597501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.595529 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf" pod "0680007d-233b-4386-bb5c-6f6595f2bdab" (UID: "0680007d-233b-4386-bb5c-6f6595f2bdab"). InnerVolumeSpecName "kube-api-access-tdmtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:55.693862 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693839 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0680007d-233b-4386-bb5c-6f6595f2bdab-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693868 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693881 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693895 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693908 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693919 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693934 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmtf\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-kube-api-access-tdmtf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693946 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.694016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.693959 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0680007d-233b-4386-bb5c-6f6595f2bdab-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:55.767464 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.767373 5011 generic.go:356] "Generic (PLEG): container finished" podID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerID="b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd" exitCode=0 Oct 31 00:54:55.767635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.767463 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" Oct 31 00:54:55.767635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.767477 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" event={"ID":"0680007d-233b-4386-bb5c-6f6595f2bdab","Type":"ContainerDied","Data":"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd"} Oct 31 00:54:55.767635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.767542 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm" event={"ID":"0680007d-233b-4386-bb5c-6f6595f2bdab","Type":"ContainerDied","Data":"4be7de629b32a109686a3dda3030854d21097fccccdbc14feacb5773cb38edc1"} Oct 31 00:54:55.767635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.767575 5011 scope.go:122] "RemoveContainer" containerID="b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd" Oct 31 00:54:55.855257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.855099 5011 scope.go:122] "RemoveContainer" containerID="b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd" Oct 31 00:54:55.856288 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:54:55.855844 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd\": container with ID starting with b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd not found: ID does not exist" containerID="b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd" Oct 31 00:54:55.856288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.855889 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd"} err="failed to get container status \"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd\": rpc error: code = NotFound desc = could not find container \"b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd\": container with ID starting with b7b854f001f44dad92ddeb69bb8a3433e9c12cb243fab85c413078ea70104dfd not found: ID does not exist" Oct 31 00:54:55.872245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.870801 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:54:55.941232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:55.923580 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-nc7dm"] Oct 31 00:54:56.325947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.325918 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:54:56.412698 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.412658 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.412890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.412757 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.412890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.412795 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.412890 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.412875 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.413131 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.412932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.413131 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413035 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.413277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413133 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.413277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413165 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.413393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413285 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.413393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413341 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\") pod \"d5c8d202-c895-41f8-8204-b3f99251e1cf\" (UID: \"d5c8d202-c895-41f8-8204-b3f99251e1cf\") " Oct 31 00:54:56.414237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.413831 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.414237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.414121 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:56.414237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.414178 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.414628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.414578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.417146 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.417173 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.417187 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.417202 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.417264 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c8d202-c895-41f8-8204-b3f99251e1cf-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.430229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.428656 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.434392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.432633 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.434392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.432705 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:56.434392 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.432855 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8" pod "d5c8d202-c895-41f8-8204-b3f99251e1cf" (UID: "d5c8d202-c895-41f8-8204-b3f99251e1cf"). InnerVolumeSpecName "kube-api-access-tx5s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.519197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.519165 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.519197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.519192 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.519197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.519201 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx5s8\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-kube-api-access-tx5s8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.519197 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.519231 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c8d202-c895-41f8-8204-b3f99251e1cf-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.847826 5011 generic.go:356] "Generic (PLEG): container finished" podID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerID="4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd" exitCode=0 Oct 31 00:54:56.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.848016 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" event={"ID":"d5c8d202-c895-41f8-8204-b3f99251e1cf","Type":"ContainerDied","Data":"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd"} Oct 31 00:54:56.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.848068 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" event={"ID":"d5c8d202-c895-41f8-8204-b3f99251e1cf","Type":"ContainerDied","Data":"6aa189b2e1ba2363208072dc98ddeabfa6dd280f278d0d3206f1b760441d7814"} Oct 31 00:54:56.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.848102 5011 scope.go:122] "RemoveContainer" containerID="4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd" Oct 31 00:54:56.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.848237 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42" Oct 31 00:54:56.921117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.920023 5011 scope.go:122] "RemoveContainer" containerID="4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd" Oct 31 00:54:56.921117 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:54:56.920415 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd\": container with ID starting with 4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd not found: ID does not exist" containerID="4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd" Oct 31 00:54:56.921117 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.920463 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd"} err="failed to get container status \"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd\": rpc error: code = NotFound desc = could not find container \"4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd\": container with ID starting with 4668c848650d7d06f5ec0a6a4a02934e2e50df54366e265d0d622ebe580a86bd not found: ID does not exist" Oct 31 00:54:56.952387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.951929 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:54:56.964288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:56.964252 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-z5s42"] Oct 31 00:54:57.686475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:57.686442 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" path="/var/lib/kubelet/pods/0680007d-233b-4386-bb5c-6f6595f2bdab/volumes" Oct 31 00:54:57.687009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:57.686994 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" path="/var/lib/kubelet/pods/d5c8d202-c895-41f8-8204-b3f99251e1cf/volumes" Oct 31 00:54:59.197942 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.197905 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 00:54:59.198797 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198773 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" Oct 31 00:54:59.198797 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198797 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" Oct 31 00:54:59.198957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198822 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" containerName="revoke-keys" Oct 31 00:54:59.198957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198829 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" containerName="revoke-keys" Oct 31 00:54:59.198957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198857 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" Oct 31 00:54:59.198957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.198865 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" Oct 31 00:54:59.199151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.199030 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c8d202-c895-41f8-8204-b3f99251e1cf" containerName="istio-proxy" Oct 31 00:54:59.199151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.199047 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" containerName="revoke-keys" Oct 31 00:54:59.199151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.199061 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0680007d-233b-4386-bb5c-6f6595f2bdab" containerName="istio-proxy" Oct 31 00:54:59.204305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.204283 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.224048 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.224018 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 00:54:59.230827 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.230794 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-609843\"/\"kube-root-ca.crt\"" Oct 31 00:54:59.230946 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.230794 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-609843\"/\"default-dockercfg-6mdg7\"" Oct 31 00:54:59.233514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.233483 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-609843\"/\"openshift-service-ca.crt\"" Oct 31 00:54:59.233640 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.233579 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-609843\"/\"test-model-609843-kserve-self-signed-certs\"" Oct 31 00:54:59.315000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.314968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.315138 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.315093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lprr\" (UniqueName: \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.315220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.315195 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.315304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.315288 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.315346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.315316 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.315346 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.315339 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417024 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.416987 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417080 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lprr\" (UniqueName: \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417273 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417181 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417314 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417347 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417381 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.417941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417909 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.418071 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.417937 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.418152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.418083 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.420908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.420282 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.420908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.420838 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.449151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.444109 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lprr\" (UniqueName: \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\") pod \"test-model-609843-kserve-649cd84dff-77r45\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.542263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.541234 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:54:59.943376 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:54:59.943345 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 00:55:00.903175 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:00.903135 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" event={"ID":"a066ab6d-e8ed-44bc-addc-b0e5859462d5","Type":"ContainerStarted","Data":"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93"} Oct 31 00:55:00.903572 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:00.903187 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" event={"ID":"a066ab6d-e8ed-44bc-addc-b0e5859462d5","Type":"ContainerStarted","Data":"4d54d2d2454b5e564fb4566a6cdb2acbb313546e7ef939b4cc1bafdcb9caba5a"} Oct 31 00:55:00.947752 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:00.947703 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" podStartSLOduration=1.9476866689999999 podStartE2EDuration="1.947686669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:00.945133458 +0000 UTC m=+2169.394221098" watchObservedRunningTime="2026-10-31 00:55:00.947686669 +0000 UTC m=+2169.396774309" Oct 31 00:55:01.407253 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.407199 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 00:55:01.412531 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.412507 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.453695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.444374 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72120c-openshift-default-dockercfg-kc6sd\"" Oct 31 00:55:01.457723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.457692 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 00:55:01.495077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.495041 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.495360 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.495335 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.495619 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.495582 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.495804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.495787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9g7\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.495933 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.495920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.496039 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.496027 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.496364 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.496291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.496523 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.496484 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.496699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.496677 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.561237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.552786 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600697 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600744 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9g7\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600774 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600793 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.600927 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.601000 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.601041 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.601073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.601518 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.607516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.612288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.607741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.624144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.623550 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.624144 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.623783 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.636180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.636148 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.679279 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.663770 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.735775 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.735739 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.799230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:01.781973 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9g7\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\") pod \"e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:01.954253 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:55:01.954127 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:55:02.031454 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.031421 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:02.369473 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.369439 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 00:55:02.370402 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:55:02.370370 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8016e34f_8e5f_4f34_865e_ce8b044c3ff6.slice/crio-01a4602826c898743d7007876c1e616b56d4392d4228c700f86061780bb95ce5 WatchSource:0}: Error finding container 01a4602826c898743d7007876c1e616b56d4392d4228c700f86061780bb95ce5: Status 404 returned error can't find the container with id 01a4602826c898743d7007876c1e616b56d4392d4228c700f86061780bb95ce5 Oct 31 00:55:02.373544 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.373510 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:02.373635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.373581 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:02.373635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.373616 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:02.940563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.940523 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerStarted","Data":"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945"} Oct 31 00:55:02.940880 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:02.940586 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerStarted","Data":"01a4602826c898743d7007876c1e616b56d4392d4228c700f86061780bb95ce5"} Oct 31 00:55:03.026082 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:03.026015 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podStartSLOduration=2.025996663 podStartE2EDuration="2.025996663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:03.02582943 +0000 UTC m=+2171.474917073" watchObservedRunningTime="2026-10-31 00:55:03.025996663 +0000 UTC m=+2171.475084302" Oct 31 00:55:03.033329 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:03.033305 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:03.040544 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:03.040521 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:03.951824 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:03.951798 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:03.953266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:03.952985 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:55:09.541801 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:09.541764 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:55:09.559881 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:09.559854 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:55:10.030791 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:10.030756 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 00:55:11.782723 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.782687 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 00:55:11.788391 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.788369 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.830228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.819620 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d5621-openshift-default-dockercfg-hn752\"" Oct 31 00:55:11.891229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891144 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891332 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891416 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891533 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891467 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbs5\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891544 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891628 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891605 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891665 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891692 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.891730 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.891720 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.944390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.944345 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.992815 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.992866 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.992898 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.992978 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993134 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993233 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993287 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbs5\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993426 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.993910 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:11.994964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.994815 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.001286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.995506 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.001286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.996425 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.001286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:11.996721 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.104234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.095055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.104234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.100501 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.113373 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.105178 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.127356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.125741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbs5\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\") pod \"e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.417950 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.417912 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:12.746850 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.746820 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 00:55:12.747178 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:55:12.747156 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad55bd2_5eae_4763_b439_c254281da697.slice/crio-8deb0bc828fb3c8801ecf03cc2d1e74e035aa1d9c3053ad0e95a316614ba4a02 WatchSource:0}: Error finding container 8deb0bc828fb3c8801ecf03cc2d1e74e035aa1d9c3053ad0e95a316614ba4a02: Status 404 returned error can't find the container with id 8deb0bc828fb3c8801ecf03cc2d1e74e035aa1d9c3053ad0e95a316614ba4a02 Oct 31 00:55:12.749869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.749834 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:12.749968 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.749948 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:12.750010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:12.749980 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:55:13.078588 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:13.078538 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" event={"ID":"7ad55bd2-5eae-4763-b439-c254281da697","Type":"ContainerStarted","Data":"05fafc6d7770f03b7db397beca6564b8bf4f6184dd364d5432e1c7281746a018"} Oct 31 00:55:13.079001 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:13.078636 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" event={"ID":"7ad55bd2-5eae-4763-b439-c254281da697","Type":"ContainerStarted","Data":"8deb0bc828fb3c8801ecf03cc2d1e74e035aa1d9c3053ad0e95a316614ba4a02"} Oct 31 00:55:13.423584 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:13.423490 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:13.424232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:13.423889 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:13.424232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:13.423941 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:14.422229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:14.421556 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:14.422229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:14.421617 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:15.423231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:15.421540 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:15.423231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:15.421617 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:16.427232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:16.426277 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:16.427232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:16.426336 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:17.428231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:17.426498 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:17.428231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:17.426579 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:18.425233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:18.423601 5011 patch_prober.go:29] interesting pod/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 00:55:18.425233 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:18.423667 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 00:55:19.425463 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:19.425428 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:19.425956 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:19.425933 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:19.426639 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:19.426616 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 00:55:19.486835 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:19.486790 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podStartSLOduration=8.486774394 podStartE2EDuration="8.486774394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:13.146614743 +0000 UTC m=+2181.595702385" watchObservedRunningTime="2026-10-31 00:55:19.486774394 +0000 UTC m=+2187.935862033" Oct 31 00:55:20.561250 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:20.561195 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:55:20.561756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:20.561459 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" containerID="cri-o://a4285f9f76a02ae66f5182ea42cce23e0be7e74b00b13a754d9fce5948bf5d28" gracePeriod=30 Oct 31 00:55:25.799642 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:25.799591 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l"] Oct 31 00:55:25.806199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:25.806164 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbb902-xhr9l"] Oct 31 00:55:26.270420 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.270391 5011 generic.go:356] "Generic (PLEG): container finished" podID="907a0c69-256f-40d9-b3d7-ea093bece237" containerID="a4285f9f76a02ae66f5182ea42cce23e0be7e74b00b13a754d9fce5948bf5d28" exitCode=0 Oct 31 00:55:26.270564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.270453 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" event={"ID":"907a0c69-256f-40d9-b3d7-ea093bece237","Type":"ContainerDied","Data":"a4285f9f76a02ae66f5182ea42cce23e0be7e74b00b13a754d9fce5948bf5d28"} Oct 31 00:55:26.270564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.270494 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" event={"ID":"907a0c69-256f-40d9-b3d7-ea093bece237","Type":"ContainerDied","Data":"2b72cb04f055e418569001db34a8b324e64abbae825067b07a3154e639e5502e"} Oct 31 00:55:26.270564 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.270509 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b72cb04f055e418569001db34a8b324e64abbae825067b07a3154e639e5502e" Oct 31 00:55:26.289453 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.289428 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:55:26.401300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401179 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401277 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401486 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401326 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401486 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401392 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401486 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401421 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401486 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401466 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401485 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401513 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401561 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401561 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401652 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\") pod \"907a0c69-256f-40d9-b3d7-ea093bece237\" (UID: \"907a0c69-256f-40d9-b3d7-ea093bece237\") " Oct 31 00:55:26.401697 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401684 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:26.402019 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.401783 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:26.402172 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402142 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:26.402513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402491 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.402629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402516 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.402629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402531 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.402629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402544 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907a0c69-256f-40d9-b3d7-ea093bece237-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.402629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.402556 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.403756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.403733 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:26.403887 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.403871 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:26.404014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.403996 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:26.404146 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.404123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp" pod "907a0c69-256f-40d9-b3d7-ea093bece237" (UID: "907a0c69-256f-40d9-b3d7-ea093bece237"). InnerVolumeSpecName "kube-api-access-nh8tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:26.504541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.504509 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907a0c69-256f-40d9-b3d7-ea093bece237-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.504541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.504538 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-kube-api-access-nh8tp\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.504541 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.504549 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907a0c69-256f-40d9-b3d7-ea093bece237-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:26.504777 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:26.504558 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907a0c69-256f-40d9-b3d7-ea093bece237-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:27.278287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:27.278255 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb" Oct 31 00:55:27.345892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:27.345864 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:55:27.358580 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:27.358546 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-zv6bb"] Oct 31 00:55:27.685825 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:27.685789 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45119cea-00e4-4a72-8dbe-0261215d0f34" path="/var/lib/kubelet/pods/45119cea-00e4-4a72-8dbe-0261215d0f34/volumes" Oct 31 00:55:27.686417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:27.686395 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" path="/var/lib/kubelet/pods/907a0c69-256f-40d9-b3d7-ea093bece237/volumes" Oct 31 00:55:31.948326 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:55:31.948284 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:55:36.672478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.672395 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 00:55:36.673096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.673079 5011 cpu_manager.go:414] "containerMap: removing container" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" Oct 31 00:55:36.673160 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.673097 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" Oct 31 00:55:36.673260 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.673249 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="907a0c69-256f-40d9-b3d7-ea093bece237" containerName="istio-proxy" Oct 31 00:55:36.678488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.678461 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.699025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.699002 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 00:55:36.701590 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.701567 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-01a117-serving-cert\"" Oct 31 00:55:36.782452 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.782414 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcbz\" (UniqueName: \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.782633 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.782555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.782683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.782646 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.782683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.782668 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.884990 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.884947 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.885282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.885259 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.885439 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.885413 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.885809 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.885783 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcbz\" (UniqueName: \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.886325 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.886265 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.889870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.889847 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.892860 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.892837 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.917278 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.917197 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcbz\" (UniqueName: \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\") pod \"maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:36.995838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:36.995454 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:37.316177 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:55:37.316148 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f0ca78_8042_4926_8e87_69510deea04d.slice/crio-1a6e5f8df50d85d1f48b65d70ef15aba28c202854ee67c9e6707a776564092d7 WatchSource:0}: Error finding container 1a6e5f8df50d85d1f48b65d70ef15aba28c202854ee67c9e6707a776564092d7: Status 404 returned error can't find the container with id 1a6e5f8df50d85d1f48b65d70ef15aba28c202854ee67c9e6707a776564092d7 Oct 31 00:55:37.316603 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:37.316573 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 00:55:37.365187 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:37.365152 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" event={"ID":"38f0ca78-8042-4926-8e87-69510deea04d","Type":"ContainerStarted","Data":"1a6e5f8df50d85d1f48b65d70ef15aba28c202854ee67c9e6707a776564092d7"} Oct 31 00:55:38.386921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:38.386883 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" event={"ID":"38f0ca78-8042-4926-8e87-69510deea04d","Type":"ContainerStarted","Data":"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b"} Oct 31 00:55:38.387433 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:38.386957 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:38.431257 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:38.431163 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" podStartSLOduration=1.963368208 podStartE2EDuration="2.431143718s" totalImagesPullingTime="467.77551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:38.429775277 +0000 UTC m=+2206.878862918" watchObservedRunningTime="2026-10-31 00:55:38.431143718 +0000 UTC m=+2206.880231358" Oct 31 00:55:44.402771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:44.402742 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 00:55:50.125875 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:50.125831 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs"] Oct 31 00:55:50.126477 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:50.126456 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82d887-5rlgs"] Oct 31 00:55:50.764842 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:50.764805 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:55:50.765163 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:50.765120 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" containerID="cri-o://c228936e21121cc0fc5404ab92f095eb042bcecc08b37f9b00668c3ce1eae446" gracePeriod=30 Oct 31 00:55:51.684124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:51.684090 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eab3186-6f90-4399-b6d1-21c54c57fea6" path="/var/lib/kubelet/pods/8eab3186-6f90-4399-b6d1-21c54c57fea6/volumes" Oct 31 00:55:52.927661 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:52.927633 5011 scope.go:122] "RemoveContainer" containerID="fcc26afcab26044f1bfc9d4b4f8a27db65280882c4af8f9242c2e4e469349c44" Oct 31 00:55:52.939722 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:52.939697 5011 scope.go:122] "RemoveContainer" containerID="6b7ff9875c1c02fc151f77ed967a668e5e91b26a5800ee1e27fdfff6523a99cc" Oct 31 00:55:52.950695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:52.950667 5011 scope.go:122] "RemoveContainer" containerID="d78f5d24b6436bd2ac63cd1c9fd392e233e49db363c1aeece9d448e123982762" Oct 31 00:55:52.959604 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:52.959580 5011 scope.go:122] "RemoveContainer" containerID="811573f81bbee21ef524d424d76a71516cda461c1c2a0860e85af3e87d818b30" Oct 31 00:55:52.970331 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:52.970313 5011 scope.go:122] "RemoveContainer" containerID="a4285f9f76a02ae66f5182ea42cce23e0be7e74b00b13a754d9fce5948bf5d28" Oct 31 00:55:54.155251 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:54.155197 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:55:56.213712 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.213621 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:55:56.214310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.214279 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 00:55:56.643079 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.623058 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:55:56.743571 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.743532 5011 generic.go:356] "Generic (PLEG): container finished" podID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerID="c228936e21121cc0fc5404ab92f095eb042bcecc08b37f9b00668c3ce1eae446" exitCode=0 Oct 31 00:55:56.744007 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.743978 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" event={"ID":"24ae3679-1fd1-44eb-bf75-568e859767c8","Type":"ContainerDied","Data":"c228936e21121cc0fc5404ab92f095eb042bcecc08b37f9b00668c3ce1eae446"} Oct 31 00:55:56.793736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.785232 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:55:56.944283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.943969 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.944283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.944050 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.944283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.944111 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948339 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948414 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948475 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948520 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948568 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.966230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.948652 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\") pod \"24ae3679-1fd1-44eb-bf75-568e859767c8\" (UID: \"24ae3679-1fd1-44eb-bf75-568e859767c8\") " Oct 31 00:55:56.974238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.972463 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:56.974238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.973592 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.974238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.973791 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.974238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.973964 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.987283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.979289 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.994772 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995362 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ae3679-1fd1-44eb-bf75-568e859767c8-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995387 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995405 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995419 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995433 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.995472 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.995445 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:56.996139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:56.996103 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.015245 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.014406 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:57.015657 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.015633 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8" pod "24ae3679-1fd1-44eb-bf75-568e859767c8" (UID: "24ae3679-1fd1-44eb-bf75-568e859767c8"). InnerVolumeSpecName "kube-api-access-4rbq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.119267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.117198 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbq8\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-kube-api-access-4rbq8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:57.119267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.117285 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:57.121312 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.120272 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ae3679-1fd1-44eb-bf75-568e859767c8-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:55:57.779288 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.779246 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" event={"ID":"24ae3679-1fd1-44eb-bf75-568e859767c8","Type":"ContainerDied","Data":"2ffcf5dc6cf311a66899b9bf5a83ed8883491342a56b79f8b30811633c05376c"} Oct 31 00:55:57.780685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.779331 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln" Oct 31 00:55:57.780685 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.779963 5011 scope.go:122] "RemoveContainer" containerID="c228936e21121cc0fc5404ab92f095eb042bcecc08b37f9b00668c3ce1eae446" Oct 31 00:55:57.849110 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.849081 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:55:57.860302 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:57.860275 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-fjjln"] Oct 31 00:55:58.266240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:58.266172 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:55:58.267348 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:58.267085 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" containerID="cri-o://3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915" gracePeriod=30 Oct 31 00:55:58.308680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:58.307369 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:55:58.308680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:58.307595 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" containerID="cri-o://e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa" gracePeriod=30 Oct 31 00:55:59.684669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:55:59.684635 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" path="/var/lib/kubelet/pods/24ae3679-1fd1-44eb-bf75-568e859767c8/volumes" Oct 31 00:56:01.955173 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:56:01.955132 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:56:03.629658 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.629626 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:56:03.677563 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.677540 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:56:03.705478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705449 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705507 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705651 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705573 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705649 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705776 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705682 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705868 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.705947 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705916 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.706006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.705962 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.706006 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.706001 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\") pod \"3430a600-4381-4f5f-b969-579dbad40cdf\" (UID: \"3430a600-4381-4f5f-b969-579dbad40cdf\") " Oct 31 00:56:03.706669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.706579 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.706898 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.706854 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.707240 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.707194 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.707553 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.707513 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.707744 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.707719 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.709908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.709883 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.710731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.710710 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.712863 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.712834 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "kube-api-access-nl2w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.712973 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.712900 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token" pod "3430a600-4381-4f5f-b969-579dbad40cdf" (UID: "3430a600-4381-4f5f-b969-579dbad40cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.807963 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.807934 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808091 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808138 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808159 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808225 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808252 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808282 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.808521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808350 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808395 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\") pod \"c4694b61-13ca-470c-8139-0d07c715f633\" (UID: \"c4694b61-13ca-470c-8139-0d07c715f633\") " Oct 31 00:56:03.808521 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808439 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.808682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808592 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.808826 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808806 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.808912 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.808886 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.809234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809180 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3430a600-4381-4f5f-b969-579dbad40cdf-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809225 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809243 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4694b61-13ca-470c-8139-0d07c715f633-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809258 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809271 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809286 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809300 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2w9\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-kube-api-access-nl2w9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809314 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809327 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809339 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3430a600-4381-4f5f-b969-579dbad40cdf-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809356 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809351 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3430a600-4381-4f5f-b969-579dbad40cdf-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809362 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809371 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.809622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.809379 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3430a600-4381-4f5f-b969-579dbad40cdf-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.810813 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.810790 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.810889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.810852 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.810961 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.810942 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.810997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.810944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2" pod "c4694b61-13ca-470c-8139-0d07c715f633" (UID: "c4694b61-13ca-470c-8139-0d07c715f633"). InnerVolumeSpecName "kube-api-access-qcnp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.838193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.838124 5011 generic.go:356] "Generic (PLEG): container finished" podID="c4694b61-13ca-470c-8139-0d07c715f633" containerID="e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa" exitCode=0 Oct 31 00:56:03.838193 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.838167 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" Oct 31 00:56:03.838389 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.838200 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" event={"ID":"c4694b61-13ca-470c-8139-0d07c715f633","Type":"ContainerDied","Data":"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa"} Oct 31 00:56:03.838389 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.838274 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck" event={"ID":"c4694b61-13ca-470c-8139-0d07c715f633","Type":"ContainerDied","Data":"09d36ccaf69f050a079f1f8c141a71d52fe527aee5f21d6c0c78b888eba9c51b"} Oct 31 00:56:03.838389 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.838306 5011 scope.go:122] "RemoveContainer" containerID="e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa" Oct 31 00:56:03.840043 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.840020 5011 generic.go:356] "Generic (PLEG): container finished" podID="3430a600-4381-4f5f-b969-579dbad40cdf" containerID="3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915" exitCode=0 Oct 31 00:56:03.840149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.840064 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" event={"ID":"3430a600-4381-4f5f-b969-579dbad40cdf","Type":"ContainerDied","Data":"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915"} Oct 31 00:56:03.840149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.840070 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" Oct 31 00:56:03.840149 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.840103 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm" event={"ID":"3430a600-4381-4f5f-b969-579dbad40cdf","Type":"ContainerDied","Data":"8af035ffd6750bf2f9664cd4922e20043828241a1834e10dc57b1a2f65f13439"} Oct 31 00:56:03.856844 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.856818 5011 scope.go:122] "RemoveContainer" containerID="e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa" Oct 31 00:56:03.857162 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:56:03.857134 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa\": container with ID starting with e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa not found: ID does not exist" containerID="e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa" Oct 31 00:56:03.857300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.857175 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa"} err="failed to get container status \"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa\": rpc error: code = NotFound desc = could not find container \"e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa\": container with ID starting with e97a6f97b04b288b33ad45f6cd821192919fa57346a5f452d113869a63a389aa not found: ID does not exist" Oct 31 00:56:03.857300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.857198 5011 scope.go:122] "RemoveContainer" containerID="3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915" Oct 31 00:56:03.867672 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.867648 5011 scope.go:122] "RemoveContainer" containerID="3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915" Oct 31 00:56:03.867976 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:56:03.867958 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915\": container with ID starting with 3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915 not found: ID does not exist" containerID="3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915" Oct 31 00:56:03.868044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.867982 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915"} err="failed to get container status \"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915\": rpc error: code = NotFound desc = could not find container \"3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915\": container with ID starting with 3496bd9e7964afa5fa04c883094c65e03b27a3d3feaacf18b4fce8045144f915 not found: ID does not exist" Oct 31 00:56:03.909457 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.909432 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:56:03.910839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.910814 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4694b61-13ca-470c-8139-0d07c715f633-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.910839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.910844 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnp2\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-kube-api-access-qcnp2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.911034 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.910859 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4694b61-13ca-470c-8139-0d07c715f633-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.911034 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.910876 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4694b61-13ca-470c-8139-0d07c715f633-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.915238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.915215 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jgjck"] Oct 31 00:56:03.965785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.965757 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:56:03.977438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:03.977409 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jz7xm"] Oct 31 00:56:05.683979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:05.683945 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" path="/var/lib/kubelet/pods/3430a600-4381-4f5f-b969-579dbad40cdf/volumes" Oct 31 00:56:05.684515 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:05.684499 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4694b61-13ca-470c-8139-0d07c715f633" path="/var/lib/kubelet/pods/c4694b61-13ca-470c-8139-0d07c715f633/volumes" Oct 31 00:56:24.911539 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.911498 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 00:56:24.912513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912490 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" Oct 31 00:56:24.912513 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912512 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912536 5011 cpu_manager.go:414] "containerMap: removing container" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912542 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912576 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912582 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912701 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4694b61-13ca-470c-8139-0d07c715f633" containerName="istio-proxy" Oct 31 00:56:24.912715 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912711 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ae3679-1fd1-44eb-bf75-568e859767c8" containerName="istio-proxy" Oct 31 00:56:24.913025 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.912722 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3430a600-4381-4f5f-b969-579dbad40cdf" containerName="istio-proxy" Oct 31 00:56:24.917060 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.917041 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:24.937272 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.936927 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 00:56:24.943560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.942879 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b87c62\"/\"openshift-service-ca.crt\"" Oct 31 00:56:24.943560 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.942981 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b87c62\"/\"kube-root-ca.crt\"" Oct 31 00:56:24.950274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.946162 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b87c62\"/\"default-dockercfg-lmhd5\"" Oct 31 00:56:24.950274 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:24.946314 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b87c62\"/\"test-model-b87c62-kserve-self-signed-certs\"" Oct 31 00:56:25.078000 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.077957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.078186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.078013 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.078283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.078201 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.078334 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.078308 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94md\" (UniqueName: \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.078469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.078454 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.078518 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.078507 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187529 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187664 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94md\" (UniqueName: \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187822 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.188282 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.187896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.189021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.188731 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.189021 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.188984 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.200228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.190583 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.200228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.196609 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.200228 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.199453 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.229231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.225934 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94md\" (UniqueName: \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\") pod \"test-model-b87c62-kserve-659b46d979-x876c\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.247231 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.239500 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:25.673618 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:25.673582 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 00:56:26.041519 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:26.041470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" event={"ID":"b2316e45-72e5-4a67-af6a-4c765bf1501d","Type":"ContainerStarted","Data":"f8c4a27aa48dfbda51b66d6873325ede229e71e60f9a70fb246486d8ff8653e4"} Oct 31 00:56:26.041964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:26.041526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" event={"ID":"b2316e45-72e5-4a67-af6a-4c765bf1501d","Type":"ContainerStarted","Data":"3da56b1d718eaf6c11067b72bdba625817395e4f6cf7cfa452ccacf35e84ca7f"} Oct 31 00:56:26.089014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:26.088962 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" podStartSLOduration=2.088949072 podStartE2EDuration="2.088949072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:26.086564181 +0000 UTC m=+2254.535651823" watchObservedRunningTime="2026-10-31 00:56:26.088949072 +0000 UTC m=+2254.538036761" Oct 31 00:56:26.240738 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:26.240501 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:31.951128 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:56:31.951085 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:56:34.910921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:34.910887 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:56:34.911340 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:34.911126 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" containerID="cri-o://39b29288e9b2336d949dc9d94bacf9a8645641e3655b1f8aefe91d5051a7dd46" gracePeriod=30 Oct 31 00:56:35.240148 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:35.240052 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:35.253582 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:35.253549 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:36.172450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:36.172418 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 00:56:40.245930 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.245897 5011 generic.go:356] "Generic (PLEG): container finished" podID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerID="39b29288e9b2336d949dc9d94bacf9a8645641e3655b1f8aefe91d5051a7dd46" exitCode=0 Oct 31 00:56:40.246317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.245940 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" event={"ID":"4855c225-bec3-4f4a-a815-dbe5ba42b902","Type":"ContainerDied","Data":"39b29288e9b2336d949dc9d94bacf9a8645641e3655b1f8aefe91d5051a7dd46"} Oct 31 00:56:40.246317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.245993 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" event={"ID":"4855c225-bec3-4f4a-a815-dbe5ba42b902","Type":"ContainerDied","Data":"b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b"} Oct 31 00:56:40.246317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.246016 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b693e526f897be8878cb7580b575ac90109105206a36e24cfd6d045b0f9a737b" Oct 31 00:56:40.268417 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.268389 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:56:40.382531 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382494 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.382790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.382790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382602 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.382790 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382776 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.382978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382829 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.382978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382899 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:40.382978 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.382911 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.383155 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383021 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.383155 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383101 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.383155 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383111 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:40.383339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383172 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:40.383339 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383187 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\") pod \"4855c225-bec3-4f4a-a815-dbe5ba42b902\" (UID: \"4855c225-bec3-4f4a-a815-dbe5ba42b902\") " Oct 31 00:56:40.383562 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383524 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:40.383743 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.383718 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:40.384157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.384132 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4855c225-bec3-4f4a-a815-dbe5ba42b902-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.384294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.384162 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.384294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.384176 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.384294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.384248 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.384294 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.384264 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.386427 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.386395 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:40.386964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.386939 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:40.390959 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.390937 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "kube-api-access-sn74t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:40.392151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.392129 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy" pod "4855c225-bec3-4f4a-a815-dbe5ba42b902" (UID: "4855c225-bec3-4f4a-a815-dbe5ba42b902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:40.486190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.486102 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.486190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.486140 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.486190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.486154 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:40.486190 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:40.486166 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn74t\" (UniqueName: \"kubernetes.io/projected/4855c225-bec3-4f4a-a815-dbe5ba42b902-kube-api-access-sn74t\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:41.282229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:41.278414 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg" Oct 31 00:56:41.393394 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:41.393354 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:56:41.399943 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:41.399895 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-tftpg"] Oct 31 00:56:41.688244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:41.688192 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" path="/var/lib/kubelet/pods/4855c225-bec3-4f4a-a815-dbe5ba42b902/volumes" Oct 31 00:56:43.945622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.945575 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 00:56:43.946374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.946357 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" Oct 31 00:56:43.946446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.946376 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" Oct 31 00:56:43.946495 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.946484 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4855c225-bec3-4f4a-a815-dbe5ba42b902" containerName="istio-proxy" Oct 31 00:56:43.951295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.951258 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:43.959474 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:43.959449 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 00:56:44.045516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.045476 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-57e594-serving-cert\"" Oct 31 00:56:44.098799 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.098767 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.098976 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.098814 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.099044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.098980 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbjv\" (UniqueName: \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.099123 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.099097 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.200724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.200640 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.200724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.200676 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.200724 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.200701 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbjv\" (UniqueName: \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.201059 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.200728 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.201475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.201449 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.203286 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.203259 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.203379 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.203282 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.241264 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.239585 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbjv\" (UniqueName: \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\") pod \"maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.273997 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.273888 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:44.615409 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:44.615383 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 00:56:44.615540 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:56:44.615463 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8654b8ab_b4d0_4f9a_8481_01f663f5c234.slice/crio-19b0f924085be17fedd528957826e2b2cc30e9f23a933fead46e8c5d09ba7cb5 WatchSource:0}: Error finding container 19b0f924085be17fedd528957826e2b2cc30e9f23a933fead46e8c5d09ba7cb5: Status 404 returned error can't find the container with id 19b0f924085be17fedd528957826e2b2cc30e9f23a933fead46e8c5d09ba7cb5 Oct 31 00:56:45.482583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:45.482538 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" event={"ID":"8654b8ab-b4d0-4f9a-8481-01f663f5c234","Type":"ContainerStarted","Data":"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb"} Oct 31 00:56:45.483243 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:45.483177 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" event={"ID":"8654b8ab-b4d0-4f9a-8481-01f663f5c234","Type":"ContainerStarted","Data":"19b0f924085be17fedd528957826e2b2cc30e9f23a933fead46e8c5d09ba7cb5"} Oct 31 00:56:45.483450 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:45.483416 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:45.560817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:45.557022 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" podStartSLOduration=2.12947508 podStartE2EDuration="2.557000363s" totalImagesPullingTime="427.525283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:45.542510596 +0000 UTC m=+2273.991598237" watchObservedRunningTime="2026-10-31 00:56:45.557000363 +0000 UTC m=+2274.006088002" Oct 31 00:56:49.908256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:49.906653 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:36218->10.132.0.171:15021: read: connection reset by peer" start-of-body= Oct 31 00:56:49.908256 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:49.906732 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:36218->10.132.0.171:15021: read: connection reset by peer" Oct 31 00:56:50.528671 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.528637 5011 generic.go:356] "Generic (PLEG): container finished" podID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerID="146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945" exitCode=0 Oct 31 00:56:50.528869 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.528707 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerDied","Data":"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945"} Oct 31 00:56:50.529267 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.529247 5011 scope.go:122] "RemoveContainer" containerID="146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945" Oct 31 00:56:50.531304 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.531260 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:56:50.531408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.531353 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:56:50.531408 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:50.531400 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:56:51.316781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:51.316745 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:56:51.317199 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:51.316945 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" containerID="cri-o://56cb0391756cabdaec4aed1b2bd73c5022661c0fb5208763014992aab2d05549" gracePeriod=30 Oct 31 00:56:51.484432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:51.484393 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:56:51.484680 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:51.484651 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" containerID="cri-o://d35c1f33da86b8240ae7c7547c850e7b31ae3af38f82327d3b96fc6c8270930b" gracePeriod=30 Oct 31 00:56:51.561469 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:51.561433 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerStarted","Data":"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380"} Oct 31 00:56:52.032575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:52.032039 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:56:52.032575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:52.032161 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:56:52.032575 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:52.032222 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 00:56:52.504412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:52.504378 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 00:56:53.035623 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:53.035558 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:56:53.035820 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:53.035641 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 00:56:53.038235 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:53.037347 5011 scope.go:122] "RemoveContainer" containerID="78fb5abcf9ed77f3fa8b1eb669ee1d09959be596acaeeae7ad41e34d6afee320" Oct 31 00:56:53.056186 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:53.056156 5011 scope.go:122] "RemoveContainer" containerID="0b6efc0021b11a77bf39e7041dc221b8ac179b8aa91bc34bb4662f3112bb627f" Oct 31 00:56:53.065752 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:53.065730 5011 scope.go:122] "RemoveContainer" containerID="39b29288e9b2336d949dc9d94bacf9a8645641e3655b1f8aefe91d5051a7dd46" Oct 31 00:56:54.035617 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:54.035577 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:56:54.036189 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:54.036155 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 00:56:55.033626 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:55.033586 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:56:55.033908 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:55.033874 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 00:56:56.032591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.032543 5011 patch_prober.go:29] interesting pod/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:56:56.033051 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.032611 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 00:56:56.619922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.619878 5011 generic.go:356] "Generic (PLEG): container finished" podID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerID="d35c1f33da86b8240ae7c7547c850e7b31ae3af38f82327d3b96fc6c8270930b" exitCode=0 Oct 31 00:56:56.620075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.619980 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" event={"ID":"09bd0ace-323e-4af4-90bb-7e02686cc9a8","Type":"ContainerDied","Data":"d35c1f33da86b8240ae7c7547c850e7b31ae3af38f82327d3b96fc6c8270930b"} Oct 31 00:56:56.621898 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.621862 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" event={"ID":"62b9146d-33e4-4a0b-a3ec-8c69f051e30d","Type":"ContainerDied","Data":"56cb0391756cabdaec4aed1b2bd73c5022661c0fb5208763014992aab2d05549"} Oct 31 00:56:56.622031 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.621950 5011 generic.go:356] "Generic (PLEG): container finished" podID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerID="56cb0391756cabdaec4aed1b2bd73c5022661c0fb5208763014992aab2d05549" exitCode=0 Oct 31 00:56:56.662258 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.661284 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.707681 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.707774 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.707832 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.707928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.708006 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.708042 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.708145 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.708200 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.709957 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.708277 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\") pod \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\" (UID: \"62b9146d-33e4-4a0b-a3ec-8c69f051e30d\") " Oct 31 00:56:56.711234 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.711094 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.711892 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.711404 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.712460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.712435 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:56.716081 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.715841 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.716081 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.716057 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.721013 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.720534 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:56.727497 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.727068 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:56.728393 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.728356 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.728650 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.728619 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2" pod "62b9146d-33e4-4a0b-a3ec-8c69f051e30d" (UID: "62b9146d-33e4-4a0b-a3ec-8c69f051e30d"). InnerVolumeSpecName "kube-api-access-887h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:56.811151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811116 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811151 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811152 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811167 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811180 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887h2\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-kube-api-access-887h2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811193 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811223 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811236 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811250 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811387 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811264 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b9146d-33e4-4a0b-a3ec-8c69f051e30d-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.811756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.811520 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:56:56.912425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912346 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912425 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912399 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912520 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912608 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912664 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912661 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912695 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.912839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912752 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.912839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912774 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:56.912839 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912761 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.913036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.912874 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.913036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913003 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.913036 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913015 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.913255 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913113 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\") pod \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\" (UID: \"09bd0ace-323e-4af4-90bb-7e02686cc9a8\") " Oct 31 00:56:56.913504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913478 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.913911 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913885 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.914003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913917 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.914003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913932 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.914003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913946 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.914003 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.913960 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.915078 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.915050 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "kube-api-access-2d9m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:56.915634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.915614 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:56.915783 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.915760 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.915902 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:56.915876 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token" pod "09bd0ace-323e-4af4-90bb-7e02686cc9a8" (UID: "09bd0ace-323e-4af4-90bb-7e02686cc9a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:57.016004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.015965 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9m7\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-kube-api-access-2d9m7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:57.016004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.015996 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:57.016004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.016006 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:57.016247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.016015 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bd0ace-323e-4af4-90bb-7e02686cc9a8-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:56:57.036471 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.036440 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:56:57.036964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.036943 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:56:57.037760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.037738 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 00:56:57.634075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.633987 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" Oct 31 00:56:57.634075 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.634014 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw" event={"ID":"09bd0ace-323e-4af4-90bb-7e02686cc9a8","Type":"ContainerDied","Data":"b201c967d2b2f30034f71f651536329eea9bb1b8fd79e807d4e8f33d47519f08"} Oct 31 00:56:57.634353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.634098 5011 scope.go:122] "RemoveContainer" containerID="d35c1f33da86b8240ae7c7547c850e7b31ae3af38f82327d3b96fc6c8270930b" Oct 31 00:56:57.636332 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.636299 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" event={"ID":"62b9146d-33e4-4a0b-a3ec-8c69f051e30d","Type":"ContainerDied","Data":"e398e60f37eab459cd90c59aab30950a4d61933ff2fb07d04e4fd28088e3a8d7"} Oct 31 00:56:57.636460 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.636391 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22" Oct 31 00:56:57.648023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.648005 5011 scope.go:122] "RemoveContainer" containerID="56cb0391756cabdaec4aed1b2bd73c5022661c0fb5208763014992aab2d05549" Oct 31 00:56:57.688986 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.688961 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:56:57.700870 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.700843 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-knx22"] Oct 31 00:56:57.727926 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.727039 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:56:57.731756 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:57.731731 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-7nldw"] Oct 31 00:56:59.685220 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:59.685161 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" path="/var/lib/kubelet/pods/09bd0ace-323e-4af4-90bb-7e02686cc9a8/volumes" Oct 31 00:56:59.685854 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:56:59.685819 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" path="/var/lib/kubelet/pods/62b9146d-33e4-4a0b-a3ec-8c69f051e30d/volumes" Oct 31 00:57:00.290114 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.290074 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 00:57:00.291098 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291075 5011 cpu_manager.go:414] "containerMap: removing container" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" Oct 31 00:57:00.291248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291101 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" Oct 31 00:57:00.291248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291193 5011 cpu_manager.go:414] "containerMap: removing container" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" Oct 31 00:57:00.291248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291201 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" Oct 31 00:57:00.291404 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291389 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b9146d-33e4-4a0b-a3ec-8c69f051e30d" containerName="istio-proxy" Oct 31 00:57:00.291446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.291422 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bd0ace-323e-4af4-90bb-7e02686cc9a8" containerName="istio-proxy" Oct 31 00:57:00.297352 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.297308 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.311124 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.311100 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 00:57:00.339066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.338985 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-01a117\"/\"openshift-service-ca.crt\"" Oct 31 00:57:00.339066 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.339013 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-01a117\"/\"kube-root-ca.crt\"" Oct 31 00:57:00.339347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.339138 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-01a117\"/\"default-dockercfg-nlglt\"" Oct 31 00:57:00.339347 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.339297 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-01a117\"/\"test-model-01a117-kserve-self-signed-certs\"" Oct 31 00:57:00.360271 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.360406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360306 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.360406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360352 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclvn\" (UniqueName: \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.360583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360517 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.360583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360544 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.360583 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.360572 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469134 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469182 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469226 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469231 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469451 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469499 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.469663 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.469548 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclvn\" (UniqueName: \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.471113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.470572 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.471113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.470835 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.471868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.471846 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.480994 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.480968 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.481581 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.481555 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.493739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.493709 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclvn\" (UniqueName: \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\") pod \"test-model-01a117-kserve-84d96fcd9-d8w4c\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:00.612422 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:00.612378 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:01.011475 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:01.011443 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 00:57:01.011832 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:57:01.011605 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod157bb073_c882_4c53_9475_a8767ae5767d.slice/crio-a281e4e3e462fda953c72b610f101db9d8e168cdbebf5d7c60ec94ecbba1296a WatchSource:0}: Error finding container a281e4e3e462fda953c72b610f101db9d8e168cdbebf5d7c60ec94ecbba1296a: Status 404 returned error can't find the container with id a281e4e3e462fda953c72b610f101db9d8e168cdbebf5d7c60ec94ecbba1296a Oct 31 00:57:01.728074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:01.728036 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" event={"ID":"157bb073-c882-4c53-9475-a8767ae5767d","Type":"ContainerStarted","Data":"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e"} Oct 31 00:57:01.728074 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:01.728088 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" event={"ID":"157bb073-c882-4c53-9475-a8767ae5767d","Type":"ContainerStarted","Data":"a281e4e3e462fda953c72b610f101db9d8e168cdbebf5d7c60ec94ecbba1296a"} Oct 31 00:57:01.773437 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:01.773381 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" podStartSLOduration=1.773366744 podStartE2EDuration="1.773366744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:01.773359681 +0000 UTC m=+2290.222447320" watchObservedRunningTime="2026-10-31 00:57:01.773366744 +0000 UTC m=+2290.222454383" Oct 31 00:57:01.959023 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:57:01.958986 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:57:02.613275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:02.613241 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:03.074300 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.074267 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:57:03.074771 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.074738 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" podUID="0ed63ec6-7272-4463-b538-9c076a15155b" containerName="maas-api" containerID="cri-o://9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2" gracePeriod=30 Oct 31 00:57:03.783721 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.783677 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:57:03.784097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.783881 5011 generic.go:356] "Generic (PLEG): container finished" podID="0ed63ec6-7272-4463-b538-9c076a15155b" containerID="9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2" exitCode=0 Oct 31 00:57:03.784097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.783953 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" event={"ID":"0ed63ec6-7272-4463-b538-9c076a15155b","Type":"ContainerDied","Data":"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2"} Oct 31 00:57:03.784097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.784011 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" event={"ID":"0ed63ec6-7272-4463-b538-9c076a15155b","Type":"ContainerDied","Data":"e3e31dd3fee015b2f3d13283b85cffb0a2161ff81422f684f8937ec9b92a1b88"} Oct 31 00:57:03.784097 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.784039 5011 scope.go:122] "RemoveContainer" containerID="9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2" Oct 31 00:57:03.794338 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.794311 5011 scope.go:122] "RemoveContainer" containerID="9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2" Oct 31 00:57:03.794636 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:57:03.794616 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2\": container with ID starting with 9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2 not found: ID does not exist" containerID="9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2" Oct 31 00:57:03.794694 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.794646 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2"} err="failed to get container status \"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2\": rpc error: code = NotFound desc = could not find container \"9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2\": container with ID starting with 9ad6702162831e24d2085d922563b00951de68cf225d37cb087f22893a73abc2 not found: ID does not exist" Oct 31 00:57:03.860634 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.860600 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\") pod \"0ed63ec6-7272-4463-b538-9c076a15155b\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " Oct 31 00:57:03.861308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.861285 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\") pod \"0ed63ec6-7272-4463-b538-9c076a15155b\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " Oct 31 00:57:03.861591 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.861575 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\") pod \"0ed63ec6-7272-4463-b538-9c076a15155b\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " Oct 31 00:57:03.861794 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.861780 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\" (UniqueName: \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\") pod \"0ed63ec6-7272-4463-b538-9c076a15155b\" (UID: \"0ed63ec6-7272-4463-b538-9c076a15155b\") " Oct 31 00:57:03.863143 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.861150 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca" pod "0ed63ec6-7272-4463-b538-9c076a15155b" (UID: "0ed63ec6-7272-4463-b538-9c076a15155b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:03.869561 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.869525 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs" pod "0ed63ec6-7272-4463-b538-9c076a15155b" (UID: "0ed63ec6-7272-4463-b538-9c076a15155b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:03.870313 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.870285 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25" pod "0ed63ec6-7272-4463-b538-9c076a15155b" (UID: "0ed63ec6-7272-4463-b538-9c076a15155b"). InnerVolumeSpecName "kube-api-access-9hk25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:03.870979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.870938 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls" pod "0ed63ec6-7272-4463-b538-9c076a15155b" (UID: "0ed63ec6-7272-4463-b538-9c076a15155b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:03.964507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.964475 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:57:03.964507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.964505 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hk25\" (UniqueName: \"kubernetes.io/projected/0ed63ec6-7272-4463-b538-9c076a15155b-kube-api-access-9hk25\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:57:03.964507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.964516 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed63ec6-7272-4463-b538-9c076a15155b-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:57:03.964831 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.964528 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed63ec6-7272-4463-b538-9c076a15155b-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 00:57:03.973925 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.973891 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 00:57:03.974656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.974599 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0ed63ec6-7272-4463-b538-9c076a15155b" containerName="maas-api" Oct 31 00:57:03.974656 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.974619 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed63ec6-7272-4463-b538-9c076a15155b" containerName="maas-api" Oct 31 00:57:03.974829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.974722 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed63ec6-7272-4463-b538-9c076a15155b" containerName="maas-api" Oct 31 00:57:03.980045 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:03.980019 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.002549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.001676 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-762753\"/\"openshift-service-ca.crt\"" Oct 31 00:57:04.002549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.001806 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-762753\"/\"test-model-762753-kserve-self-signed-certs\"" Oct 31 00:57:04.002549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.001893 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-762753\"/\"kube-root-ca.crt\"" Oct 31 00:57:04.002549 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.002515 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-762753\"/\"default-dockercfg-k7tgn\"" Oct 31 00:57:04.011390 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.011364 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 00:57:04.066949 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.066909 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtls\" (UniqueName: \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.074044 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.074009 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.074419 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.074396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.074682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.074663 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.074888 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.074869 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.075953 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.075932 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181093 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtls\" (UniqueName: \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181237 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181310 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181308 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181360 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181504 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181423 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181624 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181553 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181618 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.181955 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.181928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.183861 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.183835 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.184742 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.184717 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.204333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.204308 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtls\" (UniqueName: \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\") pod \"test-model-762753-kserve-56df5c8f9f-xbkmc\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.314281 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.313469 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:04.713169 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:57:04.713138 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3951fb9f_26d6_43d2_b1af_91b2e11837ff.slice/crio-ad54b3b19dcbc9c95e12445e7c5da8ef3f92a83b9dc76a97e384137d6fd4df47 WatchSource:0}: Error finding container ad54b3b19dcbc9c95e12445e7c5da8ef3f92a83b9dc76a97e384137d6fd4df47: Status 404 returned error can't find the container with id ad54b3b19dcbc9c95e12445e7c5da8ef3f92a83b9dc76a97e384137d6fd4df47 Oct 31 00:57:04.713483 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.713460 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 00:57:04.795985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.795843 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" event={"ID":"3951fb9f-26d6-43d2-b1af-91b2e11837ff","Type":"ContainerStarted","Data":"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb"} Oct 31 00:57:04.795985 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.795911 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" event={"ID":"3951fb9f-26d6-43d2-b1af-91b2e11837ff","Type":"ContainerStarted","Data":"ad54b3b19dcbc9c95e12445e7c5da8ef3f92a83b9dc76a97e384137d6fd4df47"} Oct 31 00:57:04.796984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.796958 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl" Oct 31 00:57:04.840754 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.840704 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" podStartSLOduration=1.8406892849999998 podStartE2EDuration="1.840689285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:04.837504303 +0000 UTC m=+2293.286591941" watchObservedRunningTime="2026-10-31 00:57:04.840689285 +0000 UTC m=+2293.289776925" Oct 31 00:57:04.876238 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.876196 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:57:04.880068 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:04.880034 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdb61f-67f7df75-2hbvl"] Oct 31 00:57:05.314795 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:05.314757 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:05.683446 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:05.683410 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed63ec6-7272-4463-b538-9c076a15155b" path="/var/lib/kubelet/pods/0ed63ec6-7272-4463-b538-9c076a15155b/volumes" Oct 31 00:57:08.213595 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.211417 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 00:57:08.216133 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.216105 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.242105 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.242064 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 00:57:08.249145 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.249117 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9de7f5\"/\"kube-root-ca.crt\"" Oct 31 00:57:08.249280 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.249252 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9de7f5\"/\"openshift-service-ca.crt\"" Oct 31 00:57:08.250584 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.250561 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9de7f5\"/\"default-dockercfg-qcclw\"" Oct 31 00:57:08.250678 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.250602 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9de7f5\"/\"test-model-9de7f5-kserve-self-signed-certs\"" Oct 31 00:57:08.343710 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.343675 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.343921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.343744 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.343921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.343889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.344077 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.344056 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.344222 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.344186 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.344291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.344266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8td\" (UniqueName: \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.446413 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.446350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.446731 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.446706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8td\" (UniqueName: \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.446900 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.446884 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.447080 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.447063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.447244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.447230 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.447478 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.447453 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.460283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.448054 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.461308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.461281 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.462014 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.461989 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.493309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.486007 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.493309 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.487991 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.507936 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.503942 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8td\" (UniqueName: \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\") pod \"test-model-9de7f5-kserve-7f9559854c-rwtzm\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.550227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.543466 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:08.930063 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:08.930026 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 00:57:08.930633 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:57:08.930606 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fb42f14_e7fe_46ad_abd6_3019fd679c73.slice/crio-eaed738deb0b7a911fa08d9540f32cedf4af0a9ca4ffc0b51c0c8a6a28c433f9 WatchSource:0}: Error finding container eaed738deb0b7a911fa08d9540f32cedf4af0a9ca4ffc0b51c0c8a6a28c433f9: Status 404 returned error can't find the container with id eaed738deb0b7a911fa08d9540f32cedf4af0a9ca4ffc0b51c0c8a6a28c433f9 Oct 31 00:57:09.843488 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:09.843451 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" event={"ID":"5fb42f14-e7fe-46ad-abd6-3019fd679c73","Type":"ContainerStarted","Data":"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388"} Oct 31 00:57:09.843889 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:09.843503 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" event={"ID":"5fb42f14-e7fe-46ad-abd6-3019fd679c73","Type":"ContainerStarted","Data":"eaed738deb0b7a911fa08d9540f32cedf4af0a9ca4ffc0b51c0c8a6a28c433f9"} Oct 31 00:57:09.892739 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:09.892687 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" podStartSLOduration=1.892673732 podStartE2EDuration="1.892673732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:09.887272195 +0000 UTC m=+2298.336359851" watchObservedRunningTime="2026-10-31 00:57:09.892673732 +0000 UTC m=+2298.341761371" Oct 31 00:57:10.544087 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:10.544053 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:10.613426 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:10.613398 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:10.626677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:10.626649 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:10.866076 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:10.866043 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 00:57:11.802269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:11.802127 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:57:14.314291 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:14.314259 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:14.328194 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:14.328168 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:14.946891 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:14.946856 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 00:57:18.544101 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:18.544065 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:18.556855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:18.556830 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:18.980055 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:18.980027 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 00:57:23.225535 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.225501 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 00:57:23.230016 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.229995 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.250922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.250894 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 00:57:23.251599 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.251577 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-82b5e8-serving-cert\"" Oct 31 00:57:23.301246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.301186 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.301246 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.301242 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.301431 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.301372 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlt6\" (UniqueName: \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.301498 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.301486 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.402990 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.403226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.403268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.403357 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlt6\" (UniqueName: \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:57:23.403861 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-serving-cert: secret "maas-api-e2e-inf-b-82b5e8-serving-cert" not found Oct 31 00:57:23.404054 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:57:23.403934 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls podName:490768d5-5b79-4ed7-8620-bf21e060eb6c nodeName:}" failed. No retries permitted until 2026-10-31 00:57:23.903914586 +0000 UTC m=+2312.353002205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls") pod "maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" (UID: "490768d5-5b79-4ed7-8620-bf21e060eb6c") : secret "maas-api-e2e-inf-b-82b5e8-serving-cert" not found Oct 31 00:57:23.405682 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.404436 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.406927 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.406899 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.430179 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.430147 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlt6\" (UniqueName: \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.911004 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.910972 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:23.913503 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:23.913477 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:24.147677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:24.147642 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:24.484748 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:24.484717 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 00:57:24.486978 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:57:24.486950 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490768d5_5b79_4ed7_8620_bf21e060eb6c.slice/crio-6574d70fb6efb627f3aa9be6d45ac0d2d8a8bda01ff2138e2543f6b31abbd4ef WatchSource:0}: Error finding container 6574d70fb6efb627f3aa9be6d45ac0d2d8a8bda01ff2138e2543f6b31abbd4ef: Status 404 returned error can't find the container with id 6574d70fb6efb627f3aa9be6d45ac0d2d8a8bda01ff2138e2543f6b31abbd4ef Oct 31 00:57:25.026010 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:25.025980 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" event={"ID":"490768d5-5b79-4ed7-8620-bf21e060eb6c","Type":"ContainerStarted","Data":"6574d70fb6efb627f3aa9be6d45ac0d2d8a8bda01ff2138e2543f6b31abbd4ef"} Oct 31 00:57:26.038018 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:26.037978 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" event={"ID":"490768d5-5b79-4ed7-8620-bf21e060eb6c","Type":"ContainerStarted","Data":"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c"} Oct 31 00:57:26.038443 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:26.038048 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:26.082916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:26.082866 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" podStartSLOduration=2.595485015 podStartE2EDuration="3.082852074s" totalImagesPullingTime="487.367059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:26.08060074 +0000 UTC m=+2314.529688380" watchObservedRunningTime="2026-10-31 00:57:26.082852074 +0000 UTC m=+2314.531939712" Oct 31 00:57:31.961273 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:57:31.961184 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:57:32.052319 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:32.052282 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 00:57:43.164295 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.164259 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 00:57:43.181677 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.181648 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.199232 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.195584 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 00:57:43.213329 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.212310 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4f4e69\"" Oct 31 00:57:43.239230 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.224754 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4f4e69-dockercfg-tftgp\"" Oct 31 00:57:43.288569 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.288518 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.288802 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.288775 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m2q2\" (UniqueName: \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.390092 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.390056 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.390305 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.390244 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m2q2\" (UniqueName: \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.390828 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.390807 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.414456 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.414389 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m2q2\" (UniqueName: \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\") pod \"payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.541276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.541241 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:43.855946 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:57:43.855910 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2f39f9_3b01_4649_9623_0e90d4d8dadb.slice/crio-3cd5557cae73b7937997151bfb0f00d09160b1c7d0a180dfe983a5d334ec6762 WatchSource:0}: Error finding container 3cd5557cae73b7937997151bfb0f00d09160b1c7d0a180dfe983a5d334ec6762: Status 404 returned error can't find the container with id 3cd5557cae73b7937997151bfb0f00d09160b1c7d0a180dfe983a5d334ec6762 Oct 31 00:57:43.856147 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:43.856119 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 00:57:44.322275 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:44.322198 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" event={"ID":"ee2f39f9-3b01-4649-9623-0e90d4d8dadb","Type":"ContainerStarted","Data":"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401"} Oct 31 00:57:44.322788 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:44.322409 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:57:44.322788 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:44.322456 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" event={"ID":"ee2f39f9-3b01-4649-9623-0e90d4d8dadb","Type":"ContainerStarted","Data":"3cd5557cae73b7937997151bfb0f00d09160b1c7d0a180dfe983a5d334ec6762"} Oct 31 00:57:44.364342 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:44.364287 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" podStartSLOduration=1.364273107 podStartE2EDuration="1.364273107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:44.363476367 +0000 UTC m=+2332.812564007" watchObservedRunningTime="2026-10-31 00:57:44.364273107 +0000 UTC m=+2332.813360747" Oct 31 00:57:53.158268 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:53.158237 5011 scope.go:122] "RemoveContainer" containerID="8de09a2e3fc3ce7ed02c6ffce94baaae209131ad0cd7d2fc6547a7a2c8cefb72" Oct 31 00:57:53.172742 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:53.172702 5011 scope.go:122] "RemoveContainer" containerID="d7a953c7ac3c848070aad4369c2ad9dc47006f61fbacd42fd766f26b2eca96af" Oct 31 00:57:53.182804 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:53.182784 5011 scope.go:122] "RemoveContainer" containerID="259659bfac94e4708774cf11fa423a3cd281a3ee4907ced1016ad9b79513a98f" Oct 31 00:57:55.335736 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:57:55.335708 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 00:58:01.226690 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.226655 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 00:58:01.232898 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.232874 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.258152 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.258121 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 00:58:01.262432 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.262404 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c942c9-serving-cert\"" Oct 31 00:58:01.351333 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.351302 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.351524 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.351499 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6shw\" (UniqueName: \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.351683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.351666 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.351725 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.351711 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.453935 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.453655 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6shw\" (UniqueName: \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.454139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.454055 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.454139 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.454124 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.454317 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.454169 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.455592 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.455544 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.457747 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.457725 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.466785 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.466755 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.490699 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.490618 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6shw\" (UniqueName: \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\") pod \"maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.549486 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.549451 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:01.900227 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:01.900102 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 00:58:01.967418 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:58:01.960917 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4836cb85_6281_4d15_85ff_ffdea2e13afe.slice/crio-8e676d7732de3010b42b12b3b8971629121025f97684c345f8b7b586af68564e WatchSource:0}: Error finding container 8e676d7732de3010b42b12b3b8971629121025f97684c345f8b7b586af68564e: Status 404 returned error can't find the container with id 8e676d7732de3010b42b12b3b8971629121025f97684c345f8b7b586af68564e Oct 31 00:58:02.170284 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:58:02.161294 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:58:02.508158 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:02.508127 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" event={"ID":"4836cb85-6281-4d15-85ff-ffdea2e13afe","Type":"ContainerStarted","Data":"8e676d7732de3010b42b12b3b8971629121025f97684c345f8b7b586af68564e"} Oct 31 00:58:03.529781 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:03.529730 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" event={"ID":"4836cb85-6281-4d15-85ff-ffdea2e13afe","Type":"ContainerStarted","Data":"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d"} Oct 31 00:58:03.530244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:03.529944 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:03.578100 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:03.578042 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" podStartSLOduration=2.117585027 podStartE2EDuration="2.578025478s" totalImagesPullingTime="460.440451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:03.575077589 +0000 UTC m=+2352.024165228" watchObservedRunningTime="2026-10-31 00:58:03.578025478 +0000 UTC m=+2352.027113120" Oct 31 00:58:09.543438 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:09.543406 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 00:58:27.340037 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.340005 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 00:58:27.344635 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.344614 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.362706 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.362674 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9de7f5-openshift-default-dockercfg-h65hq\"" Oct 31 00:58:27.367023 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.366999 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 00:58:27.469480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.469244 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.469853 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.469814 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.470180 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.470142 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.470941 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.470878 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ws2r\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.471221 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.471172 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.471542 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.471508 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.471793 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.471765 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.472009 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.471979 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.472236 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.472192 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574062 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574023 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574079 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574120 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574162 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574266 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574258 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574501 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574485 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ws2r\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574532 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574569 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574582 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574629 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574601 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.574868 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.574754 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.575247 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.575198 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.575374 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.575295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.577480 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.577445 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.577691 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.577666 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.598516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.598455 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ws2r\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.598516 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.598467 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\") pod \"e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.659406 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.659376 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:27.991616 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.991572 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 00:58:27.992854 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:58:27.992819 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc85df_4267_40d4_a5d6_dac43094d75e.slice/crio-73c81ee15546602bca435030e74b93c606d59408f291e72cf25a2f72146c1960 WatchSource:0}: Error finding container 73c81ee15546602bca435030e74b93c606d59408f291e72cf25a2f72146c1960: Status 404 returned error can't find the container with id 73c81ee15546602bca435030e74b93c606d59408f291e72cf25a2f72146c1960 Oct 31 00:58:27.996838 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.996794 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:58:27.996922 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.996885 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:58:27.996964 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:27.996933 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:58:28.780367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:28.780187 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" event={"ID":"76bc85df-4267-40d4-a5d6-dac43094d75e","Type":"ContainerStarted","Data":"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4"} Oct 31 00:58:28.780367 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:28.780295 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" event={"ID":"76bc85df-4267-40d4-a5d6-dac43094d75e","Type":"ContainerStarted","Data":"73c81ee15546602bca435030e74b93c606d59408f291e72cf25a2f72146c1960"} Oct 31 00:58:28.855229 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:28.853063 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podStartSLOduration=1.853039471 podStartE2EDuration="1.853039471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:28.840634293 +0000 UTC m=+2377.289721981" watchObservedRunningTime="2026-10-31 00:58:28.853039471 +0000 UTC m=+2377.302127113" Oct 31 00:58:29.660512 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:29.660475 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:29.667248 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:29.667195 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:58:29.667414 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:29.667275 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:58:30.660050 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:30.660007 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:58:30.660461 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:30.660088 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:58:31.660388 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:31.660351 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:58:31.660759 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:31.660415 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:58:31.974440 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:58:31.974351 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:58:32.660277 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:32.660240 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:58:32.660550 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:32.660306 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:58:33.659760 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:33.659717 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 00:58:33.659975 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:33.659793 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 00:58:34.668327 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:34.668297 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:34.669141 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:34.669105 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:34.672308 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:34.672285 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 00:58:52.316361 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:52.316335 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:58:52.343704 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:52.343681 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 00:58:53.243507 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:53.243462 5011 scope.go:122] "RemoveContainer" containerID="f97d89e6e00690cbe6bdaf6b538185c8d531d92f98f040685423306e94b76998" Oct 31 00:58:54.872299 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:58:54.872260 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 00:59:01.949362 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:59:01.949326 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:59:20.868244 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.868198 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 00:59:20.872695 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.872674 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.897237 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.897173 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 00:59:20.902983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.902958 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-57e594\"/\"default-dockercfg-jh5vs\"" Oct 31 00:59:20.903137 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.903056 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-57e594\"/\"test-model-57e594-kserve-self-signed-certs\"" Oct 31 00:59:20.925878 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.925858 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-57e594\"/\"openshift-service-ca.crt\"" Oct 31 00:59:20.925983 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.925960 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-57e594\"/\"kube-root-ca.crt\"" Oct 31 00:59:20.928717 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.928697 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.928901 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.928885 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.928979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.928911 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.928979 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.928947 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6cw\" (UniqueName: \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.929157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.929133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:20.929283 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:20.929265 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.030576 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.030537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.030778 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.030672 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.030855 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.030837 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.030916 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.030873 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.030984 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.030966 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6cw\" (UniqueName: \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.031298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.031061 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.031298 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.031116 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.031494 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.031354 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.031608 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.031586 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.033096 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.033069 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.033622 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.033604 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.055666 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.055638 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6cw\" (UniqueName: \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\") pod \"test-model-57e594-kserve-55bb9477-pkld2\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.208276 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.206008 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:21.619876 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:21.619847 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 00:59:21.622764 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:59:21.622733 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8524ac1c_f5d3_438e_9db7_61a03d95eddb.slice/crio-63145084195b8e0f4c5c0ef27dc76e5b83748463c6cc113d73353cfbcd68d01e WatchSource:0}: Error finding container 63145084195b8e0f4c5c0ef27dc76e5b83748463c6cc113d73353cfbcd68d01e: Status 404 returned error can't find the container with id 63145084195b8e0f4c5c0ef27dc76e5b83748463c6cc113d73353cfbcd68d01e Oct 31 00:59:22.397412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:22.397378 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" event={"ID":"8524ac1c-f5d3-438e-9db7-61a03d95eddb","Type":"ContainerStarted","Data":"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007"} Oct 31 00:59:22.397412 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:22.397422 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" event={"ID":"8524ac1c-f5d3-438e-9db7-61a03d95eddb","Type":"ContainerStarted","Data":"63145084195b8e0f4c5c0ef27dc76e5b83748463c6cc113d73353cfbcd68d01e"} Oct 31 00:59:22.454641 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:22.454578 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" podStartSLOduration=2.454553664 podStartE2EDuration="2.454553664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:22.449459269 +0000 UTC m=+2430.898546909" watchObservedRunningTime="2026-10-31 00:59:22.454553664 +0000 UTC m=+2430.903641304" Oct 31 00:59:23.207137 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:23.207098 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:31.206977 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:31.206944 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:31.222669 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:31.222643 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:31.494755 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:31.494678 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 00:59:31.949374 ip-10-0-132-66 kubenswrapper[5011]: E1031 00:59:31.949334 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 00:59:53.282829 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:53.282795 5011 scope.go:122] "RemoveContainer" containerID="6aeb29c084e9cb3426dad74bb91a383dab689bd39fac418fbd3b1c046715f891" Oct 31 00:59:54.032649 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:54.032618 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh"] Oct 31 00:59:54.047089 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:54.047057 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbd505-7rrnh"] Oct 31 00:59:55.691440 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:55.691398 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8" path="/var/lib/kubelet/pods/eb5b0eb3-20fb-4cf7-b1c9-3f819c804ee8/volumes" Oct 31 00:59:56.718885 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.718848 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 00:59:56.725011 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.724988 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.751113 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.751079 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f4e69\"/\"kube-root-ca.crt\"" Oct 31 00:59:56.751287 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.751181 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f4e69\"/\"openshift-service-ca.crt\"" Oct 31 00:59:56.753353 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.753327 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 00:59:56.753530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.753387 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f4e69\"/\"test-model-4f4e69-kserve-self-signed-certs\"" Oct 31 00:59:56.753530 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.753452 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f4e69\"/\"default-dockercfg-66w98\"" Oct 31 00:59:56.809921 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.809889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.810156 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.810130 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.810311 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.810294 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.810462 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.810438 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n5l\" (UniqueName: \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.810567 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.810541 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.810632 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.810600 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.912514 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912479 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.912683 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912565 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.912757 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912704 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n5l\" (UniqueName: \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.912817 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912755 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.912955 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912932 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.913046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.912953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.913046 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.913021 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.913157 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.913044 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.913269 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.913184 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.915263 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.915197 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.915505 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.915483 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:56.936335 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:56.936306 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n5l\" (UniqueName: \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\") pod \"test-model-4f4e69-kserve-6d655d7775-wthqs\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:57.056607 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:57.056507 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 00:59:57.525863 ip-10-0-132-66 kubenswrapper[5011]: W1031 00:59:57.525831 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ebc792_aff3_4ac8_8872_5936bf23292d.slice/crio-85625da99e60b3ab73899e8f479242a16522d663029e4b1acd203513738ac30d WatchSource:0}: Error finding container 85625da99e60b3ab73899e8f479242a16522d663029e4b1acd203513738ac30d: Status 404 returned error can't find the container with id 85625da99e60b3ab73899e8f479242a16522d663029e4b1acd203513738ac30d Oct 31 00:59:57.526784 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:57.526754 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 00:59:57.744120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:57.744080 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" event={"ID":"f6ebc792-aff3-4ac8-8872-5936bf23292d","Type":"ContainerStarted","Data":"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18"} Oct 31 00:59:57.744120 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:57.744132 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" event={"ID":"f6ebc792-aff3-4ac8-8872-5936bf23292d","Type":"ContainerStarted","Data":"85625da99e60b3ab73899e8f479242a16522d663029e4b1acd203513738ac30d"} Oct 31 00:59:57.789648 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:57.789556 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" podStartSLOduration=1.789539976 podStartE2EDuration="1.789539976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:57.786090947 +0000 UTC m=+2466.235178595" watchObservedRunningTime="2026-10-31 00:59:57.789539976 +0000 UTC m=+2466.238627620" Oct 31 00:59:58.058679 ip-10-0-132-66 kubenswrapper[5011]: I1031 00:59:58.058584 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:00:00.185031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.184992 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:00.189788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.189764 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:00.189932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.189874 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.194112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.194092 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.228410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.228381 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:00.239978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.239948 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:00.288867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.288348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.288867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.288411 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.288867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.288611 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m762\" (UniqueName: \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.288867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.288637 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hk7\" (UniqueName: \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.297309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.294132 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:00.304495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.304475 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.308464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.308428 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:00:00.319700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.319074 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:00.329651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.329631 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.339896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.339720 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:00.345753 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.345730 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.358692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.358671 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:00.362713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.362692 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:00.368741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.368724 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:00.380266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.376817 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.380266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.378037 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:00.391432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.391389 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m762\" (UniqueName: \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.391603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.391584 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hk7\" (UniqueName: \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.391927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.391885 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.392196 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.392173 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.392390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.392370 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.392664 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.392642 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmfm\" (UniqueName: \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.393080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.393058 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.393246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.393186 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:00.393323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.393261 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:00.393390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.393362 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.393715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.393693 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.397882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.397864 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:00.432919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.432883 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:00.443129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.442854 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:00.445164 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.445140 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.446016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.445897 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m762\" (UniqueName: \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\") pod \"maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.447883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.447814 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hk7\" (UniqueName: \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\") pod \"maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.495952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.495923 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:00.496321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.496301 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.497039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.496979 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnzt\" (UniqueName: \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.497175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497073 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.497175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497110 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9sm\" (UniqueName: \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.497365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497186 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmfm\" (UniqueName: \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.497365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5xw\" (UniqueName: \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.497496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497393 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.497496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497467 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.497663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497514 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.497663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497581 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.497663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.497631 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9mc\" (UniqueName: \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.499222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.499179 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.520553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.520032 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:00.521373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.520853 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:00.539682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.536682 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:00.539682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.538612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmfm\" (UniqueName: \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\") pod \"maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.598977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.598944 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.599235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.599199 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.599382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.599365 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.599534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.599517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9mc\" (UniqueName: \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.599683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.599666 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmlf\" (UniqueName: \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.599821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.599806 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.600048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600031 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnzt\" (UniqueName: \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.600193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600177 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.600352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9sm\" (UniqueName: \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.600640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600618 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hfw\" (UniqueName: \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.600791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600773 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.601644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.601620 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5xw\" (UniqueName: \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.601863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.601835 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.601948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.600082 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.602512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.602485 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.602789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.602767 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.630168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.630131 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnzt\" (UniqueName: \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.632458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.632429 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:00.665075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.664905 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5xw\" (UniqueName: \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.665075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.665030 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9mc\" (UniqueName: \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:00.672047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.671395 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9sm\" (UniqueName: \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.672987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.672774 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:00.704472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.704300 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmlf\" (UniqueName: \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.704472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.704361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.704667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.704563 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hfw\" (UniqueName: \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.704667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.704611 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.705531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.705505 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.706498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.706462 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.710882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.710670 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:00.719570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.719382 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:00.730592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.730139 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmlf\" (UniqueName: \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.736482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.735955 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hfw\" (UniqueName: \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.915065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.915036 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:00.938608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.938550 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:00.939925 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:00.939896 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b53a2a_bd43_4c27_8815_5f47bad4a011.slice/crio-ec1144731118cab1f270a4d45809d7e833abff35f4a8a9720fad345d02dd6e3b WatchSource:0}: Error finding container ec1144731118cab1f270a4d45809d7e833abff35f4a8a9720fad345d02dd6e3b: Status 404 returned error can't find the container with id ec1144731118cab1f270a4d45809d7e833abff35f4a8a9720fad345d02dd6e3b Oct 31 01:00:00.947376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.947001 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:00.959512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:00.959330 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:01.054841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.054729 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:01.061290 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:01.061154 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97a90a6_88a9_48fa_9c79_624fd9bd2b28.slice/crio-d4cfba7b97fb60abdc4d4326a6ce36686dbef0cce73e7b14ea1706f0612eccf7 WatchSource:0}: Error finding container d4cfba7b97fb60abdc4d4326a6ce36686dbef0cce73e7b14ea1706f0612eccf7: Status 404 returned error can't find the container with id d4cfba7b97fb60abdc4d4326a6ce36686dbef0cce73e7b14ea1706f0612eccf7 Oct 31 01:00:01.071547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.071520 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:01.086734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.086698 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:01.103745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.103705 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:01.158075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.158042 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:01.275252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.275200 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:01.276645 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:01.276617 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a8bd4e_7c2b_4a1b_b71f_fb8075d95aa7.slice/crio-bfcb2eb7cebdc55b3b5e32c0bc656e317cc66cb70d7e2e5b2f9eb5b559056235 WatchSource:0}: Error finding container bfcb2eb7cebdc55b3b5e32c0bc656e317cc66cb70d7e2e5b2f9eb5b559056235: Status 404 returned error can't find the container with id bfcb2eb7cebdc55b3b5e32c0bc656e317cc66cb70d7e2e5b2f9eb5b559056235 Oct 31 01:00:01.339667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.339636 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:01.351018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.350992 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:01.423859 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:01.423325 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad06da2_626f_4eea_97b1_1b47431c255c.slice/crio-fbf7b1415c704caa53a872c7ea73b2bd3adfcfc973e39dad3292dcb6c5bf00b8 WatchSource:0}: Error finding container fbf7b1415c704caa53a872c7ea73b2bd3adfcfc973e39dad3292dcb6c5bf00b8: Status 404 returned error can't find the container with id fbf7b1415c704caa53a872c7ea73b2bd3adfcfc973e39dad3292dcb6c5bf00b8 Oct 31 01:00:01.424476 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:01.424174 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea47c08_65fa_4182_951e_1e2e3295ce15.slice/crio-7ebcc9bc92700038b952e963b7b48822b7ce33917bdc5847b1b419708d297e90 WatchSource:0}: Error finding container 7ebcc9bc92700038b952e963b7b48822b7ce33917bdc5847b1b419708d297e90: Status 404 returned error can't find the container with id 7ebcc9bc92700038b952e963b7b48822b7ce33917bdc5847b1b419708d297e90 Oct 31 01:00:01.814449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.814415 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerStarted","Data":"e25f596d9835acec90de65152235e784a71a8cfbcd9ecbe88cce7c66858a86f4"} Oct 31 01:00:01.814601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.814479 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerStarted","Data":"48f34b9eee0289912e7f1822fffe641908f8215e33e3dabcb1acb7d03498a6d0"} Oct 31 01:00:01.815832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.815804 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerStarted","Data":"fbf7b1415c704caa53a872c7ea73b2bd3adfcfc973e39dad3292dcb6c5bf00b8"} Oct 31 01:00:01.820272 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.817489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerStarted","Data":"5745adaaee5939919bac6eadbeebdc7f7a99bf59770d2943789631f9e72c62c3"} Oct 31 01:00:01.820272 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.817539 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerStarted","Data":"ec1144731118cab1f270a4d45809d7e833abff35f4a8a9720fad345d02dd6e3b"} Oct 31 01:00:01.820472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.820445 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerStarted","Data":"78f4a596aeed10085be401fc9788798579732870fc34b6dfae083dc0470001e3"} Oct 31 01:00:01.820544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.820498 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerStarted","Data":"b662d2f03b41b8bf491c1c5d11943cbec4684e088e7b4db76e93ca8984a7b890"} Oct 31 01:00:01.823756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.822629 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" event={"ID":"f97a90a6-88a9-48fa-9c79-624fd9bd2b28","Type":"ContainerStarted","Data":"d4cfba7b97fb60abdc4d4326a6ce36686dbef0cce73e7b14ea1706f0612eccf7"} Oct 31 01:00:01.825936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.825912 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" event={"ID":"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90","Type":"ContainerStarted","Data":"d57b20105cdeecb44b9c8f3f8f5f5b585c4cad9ed704adaa3ff85dc77dcb074e"} Oct 31 01:00:01.827590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.827565 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerStarted","Data":"7ebcc9bc92700038b952e963b7b48822b7ce33917bdc5847b1b419708d297e90"} Oct 31 01:00:01.830900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.830872 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerStarted","Data":"c4d37410430aa7ccb58171fcee21c86bddd684cc60a5e73ee8471cc8701766b2"} Oct 31 01:00:01.830992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.830916 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerStarted","Data":"e3ae6d4e674d2e1fec91d293e72b6b78d7183db10b72451b124706922a44a462"} Oct 31 01:00:01.833257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.833188 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerStarted","Data":"bfcb2eb7cebdc55b3b5e32c0bc656e317cc66cb70d7e2e5b2f9eb5b559056235"} Oct 31 01:00:01.860057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.859998 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" podStartSLOduration=1.859982163 podStartE2EDuration="1.859982163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.855030753 +0000 UTC m=+2470.304118394" watchObservedRunningTime="2026-10-31 01:00:01.859982163 +0000 UTC m=+2470.309069802" Oct 31 01:00:01.898810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.898751 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" podStartSLOduration=1.898733881 podStartE2EDuration="1.898733881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.894075907 +0000 UTC m=+2470.343163559" watchObservedRunningTime="2026-10-31 01:00:01.898733881 +0000 UTC m=+2470.347821522" Oct 31 01:00:01.936819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.936761 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" podStartSLOduration=1.936742293 podStartE2EDuration="1.936742293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.934013118 +0000 UTC m=+2470.383100758" watchObservedRunningTime="2026-10-31 01:00:01.936742293 +0000 UTC m=+2470.385829937" Oct 31 01:00:01.963013 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:01.962977 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:00:01.979784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:01.979739 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" podStartSLOduration=1.979727217 podStartE2EDuration="1.979727217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.975410697 +0000 UTC m=+2470.424498336" watchObservedRunningTime="2026-10-31 01:00:01.979727217 +0000 UTC m=+2470.428814855" Oct 31 01:00:02.848052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.848018 5011 generic.go:356] "Generic (PLEG): container finished" podID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerID="e0026a20d55f607010e79ad7c63ae033a2fe957ce3f3b38ce98bd343d25d212f" exitCode=60 Oct 31 01:00:02.848516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.848107 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerDied","Data":"e0026a20d55f607010e79ad7c63ae033a2fe957ce3f3b38ce98bd343d25d212f"} Oct 31 01:00:02.848516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.848475 5011 scope.go:122] "RemoveContainer" containerID="e0026a20d55f607010e79ad7c63ae033a2fe957ce3f3b38ce98bd343d25d212f" Oct 31 01:00:02.851656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.851603 5011 generic.go:356] "Generic (PLEG): container finished" podID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerID="c4d37410430aa7ccb58171fcee21c86bddd684cc60a5e73ee8471cc8701766b2" exitCode=60 Oct 31 01:00:02.851781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.851703 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerDied","Data":"c4d37410430aa7ccb58171fcee21c86bddd684cc60a5e73ee8471cc8701766b2"} Oct 31 01:00:02.851957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.851937 5011 scope.go:122] "RemoveContainer" containerID="c4d37410430aa7ccb58171fcee21c86bddd684cc60a5e73ee8471cc8701766b2" Oct 31 01:00:02.854715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.854690 5011 generic.go:356] "Generic (PLEG): container finished" podID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerID="a9bc70d7e479f168271236522d790a69ee1451bdbb587336839aba13895e3b7c" exitCode=60 Oct 31 01:00:02.854828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.854774 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerDied","Data":"a9bc70d7e479f168271236522d790a69ee1451bdbb587336839aba13895e3b7c"} Oct 31 01:00:02.855031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.855013 5011 scope.go:122] "RemoveContainer" containerID="a9bc70d7e479f168271236522d790a69ee1451bdbb587336839aba13895e3b7c" Oct 31 01:00:02.857407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.857383 5011 generic.go:356] "Generic (PLEG): container finished" podID="92b95a10-5934-4538-806a-a932441beca7" containerID="e25f596d9835acec90de65152235e784a71a8cfbcd9ecbe88cce7c66858a86f4" exitCode=60 Oct 31 01:00:02.857546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.857526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerDied","Data":"e25f596d9835acec90de65152235e784a71a8cfbcd9ecbe88cce7c66858a86f4"} Oct 31 01:00:02.857740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.857721 5011 scope.go:122] "RemoveContainer" containerID="e25f596d9835acec90de65152235e784a71a8cfbcd9ecbe88cce7c66858a86f4" Oct 31 01:00:02.859800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.859774 5011 generic.go:356] "Generic (PLEG): container finished" podID="cad06da2-626f-4eea-97b1-1b47431c255c" containerID="b6057c920351d8cbd1fab52f6c0dda9ef35ced19323a2620275d49c225790f33" exitCode=60 Oct 31 01:00:02.859901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.859845 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerDied","Data":"b6057c920351d8cbd1fab52f6c0dda9ef35ced19323a2620275d49c225790f33"} Oct 31 01:00:02.860302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.860099 5011 scope.go:122] "RemoveContainer" containerID="b6057c920351d8cbd1fab52f6c0dda9ef35ced19323a2620275d49c225790f33" Oct 31 01:00:02.861946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.861921 5011 generic.go:356] "Generic (PLEG): container finished" podID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerID="5745adaaee5939919bac6eadbeebdc7f7a99bf59770d2943789631f9e72c62c3" exitCode=60 Oct 31 01:00:02.862040 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.862020 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerDied","Data":"5745adaaee5939919bac6eadbeebdc7f7a99bf59770d2943789631f9e72c62c3"} Oct 31 01:00:02.862358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.862330 5011 scope.go:122] "RemoveContainer" containerID="5745adaaee5939919bac6eadbeebdc7f7a99bf59770d2943789631f9e72c62c3" Oct 31 01:00:02.867668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.867632 5011 generic.go:356] "Generic (PLEG): container finished" podID="a641442d-72f6-4cc7-a9be-01585637a854" containerID="78f4a596aeed10085be401fc9788798579732870fc34b6dfae083dc0470001e3" exitCode=60 Oct 31 01:00:02.867776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.867732 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerDied","Data":"78f4a596aeed10085be401fc9788798579732870fc34b6dfae083dc0470001e3"} Oct 31 01:00:02.868054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.868036 5011 scope.go:122] "RemoveContainer" containerID="78f4a596aeed10085be401fc9788798579732870fc34b6dfae083dc0470001e3" Oct 31 01:00:02.869824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.869797 5011 generic.go:356] "Generic (PLEG): container finished" podID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerID="8b4def47f5568b816e6e97cf42949bf0282f035bef2ba697f5f198b22a4a808e" exitCode=60 Oct 31 01:00:02.869907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.869855 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" event={"ID":"f97a90a6-88a9-48fa-9c79-624fd9bd2b28","Type":"ContainerDied","Data":"8b4def47f5568b816e6e97cf42949bf0282f035bef2ba697f5f198b22a4a808e"} Oct 31 01:00:02.870074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.870060 5011 scope.go:122] "RemoveContainer" containerID="8b4def47f5568b816e6e97cf42949bf0282f035bef2ba697f5f198b22a4a808e" Oct 31 01:00:02.871593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.871572 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerID="fc18081c0000e60d0fc02efa29cc971313d409c41f90795dfc2db75fa61adbf0" exitCode=60 Oct 31 01:00:02.871669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.871647 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" event={"ID":"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90","Type":"ContainerDied","Data":"fc18081c0000e60d0fc02efa29cc971313d409c41f90795dfc2db75fa61adbf0"} Oct 31 01:00:02.871810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:02.871798 5011 scope.go:122] "RemoveContainer" containerID="fc18081c0000e60d0fc02efa29cc971313d409c41f90795dfc2db75fa61adbf0" Oct 31 01:00:03.881557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.881521 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerStarted","Data":"b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f"} Oct 31 01:00:03.883546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.883517 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerStarted","Data":"3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8"} Oct 31 01:00:03.885371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.885351 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerStarted","Data":"7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee"} Oct 31 01:00:03.887264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.887229 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerStarted","Data":"5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62"} Oct 31 01:00:03.889074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.889053 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerStarted","Data":"611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3"} Oct 31 01:00:03.890981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.890957 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerStarted","Data":"1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448"} Oct 31 01:00:03.892643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.892625 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerStarted","Data":"439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e"} Oct 31 01:00:03.925007 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:03.924964 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" podStartSLOduration=3.924950938 podStartE2EDuration="3.924950938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.923418874 +0000 UTC m=+2472.372506514" watchObservedRunningTime="2026-10-31 01:00:03.924950938 +0000 UTC m=+2472.374038577" Oct 31 01:00:04.052015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.051957 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" podStartSLOduration=4.05193795 podStartE2EDuration="4.05193795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:04.050943313 +0000 UTC m=+2472.500030954" watchObservedRunningTime="2026-10-31 01:00:04.05193795 +0000 UTC m=+2472.501025589" Oct 31 01:00:04.175427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.175377 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" podStartSLOduration=4.175364669 podStartE2EDuration="4.175364669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:04.174017878 +0000 UTC m=+2472.623105517" watchObservedRunningTime="2026-10-31 01:00:04.175364669 +0000 UTC m=+2472.624452354" Oct 31 01:00:04.903724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.903684 5011 generic.go:356] "Generic (PLEG): container finished" podID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerID="5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62" exitCode=60 Oct 31 01:00:04.904190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.903771 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerDied","Data":"5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62"} Oct 31 01:00:04.904190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.903862 5011 scope.go:122] "RemoveContainer" containerID="e0026a20d55f607010e79ad7c63ae033a2fe957ce3f3b38ce98bd343d25d212f" Oct 31 01:00:04.904190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.904092 5011 scope.go:122] "RemoveContainer" containerID="5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62" Oct 31 01:00:04.904504 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.904481 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr_odh-ai-gateway-infra(eea47c08-65fa-4182-951e-1e2e3295ce15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" Oct 31 01:00:04.906585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.906545 5011 generic.go:356] "Generic (PLEG): container finished" podID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerID="611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3" exitCode=60 Oct 31 01:00:04.906675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.906604 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerDied","Data":"611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3"} Oct 31 01:00:04.906934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.906912 5011 scope.go:122] "RemoveContainer" containerID="611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3" Oct 31 01:00:04.907246 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.907195 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw_odh-ai-gateway-infra(45849c90-dd35-4665-8f92-df1e3a20aed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" Oct 31 01:00:04.908813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.908784 5011 generic.go:356] "Generic (PLEG): container finished" podID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerID="1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448" exitCode=60 Oct 31 01:00:04.908910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.908887 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerDied","Data":"1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448"} Oct 31 01:00:04.909097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.909082 5011 scope.go:122] "RemoveContainer" containerID="1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448" Oct 31 01:00:04.909379 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.909360 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg_odh-ai-gateway-infra(93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" Oct 31 01:00:04.910938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.910919 5011 generic.go:356] "Generic (PLEG): container finished" podID="92b95a10-5934-4538-806a-a932441beca7" containerID="439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e" exitCode=60 Oct 31 01:00:04.911045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.910997 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerDied","Data":"439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e"} Oct 31 01:00:04.911243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.911187 5011 scope.go:122] "RemoveContainer" containerID="439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e" Oct 31 01:00:04.911449 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.911429 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l_odh-ai-gateway-infra(92b95a10-5934-4538-806a-a932441beca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" podUID="92b95a10-5934-4538-806a-a932441beca7" Oct 31 01:00:04.914408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.914387 5011 generic.go:356] "Generic (PLEG): container finished" podID="cad06da2-626f-4eea-97b1-1b47431c255c" containerID="b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f" exitCode=60 Oct 31 01:00:04.914527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.914454 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerDied","Data":"b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f"} Oct 31 01:00:04.914691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.914673 5011 scope.go:122] "RemoveContainer" containerID="b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f" Oct 31 01:00:04.914924 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.914904 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f_odh-ai-gateway-infra(cad06da2-626f-4eea-97b1-1b47431c255c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" Oct 31 01:00:04.916485 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.916467 5011 generic.go:356] "Generic (PLEG): container finished" podID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerID="3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8" exitCode=60 Oct 31 01:00:04.916635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.916550 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerDied","Data":"3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8"} Oct 31 01:00:04.916803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.916788 5011 scope.go:122] "RemoveContainer" containerID="3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8" Oct 31 01:00:04.917051 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.917031 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc_odh-ai-gateway-infra(06b53a2a-bd43-4c27-8815-5f47bad4a011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" Oct 31 01:00:04.919958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.919937 5011 scope.go:122] "RemoveContainer" containerID="c4d37410430aa7ccb58171fcee21c86bddd684cc60a5e73ee8471cc8701766b2" Oct 31 01:00:04.922863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.922841 5011 generic.go:356] "Generic (PLEG): container finished" podID="a641442d-72f6-4cc7-a9be-01585637a854" containerID="7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee" exitCode=60 Oct 31 01:00:04.922965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.922870 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerDied","Data":"7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee"} Oct 31 01:00:04.923133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.923116 5011 scope.go:122] "RemoveContainer" containerID="7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee" Oct 31 01:00:04.923407 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.923387 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96_odh-ai-gateway-infra(a641442d-72f6-4cc7-a9be-01585637a854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" podUID="a641442d-72f6-4cc7-a9be-01585637a854" Oct 31 01:00:04.925177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.925156 5011 generic.go:356] "Generic (PLEG): container finished" podID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerID="1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff" exitCode=60 Oct 31 01:00:04.925316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.925235 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" event={"ID":"f97a90a6-88a9-48fa-9c79-624fd9bd2b28","Type":"ContainerDied","Data":"1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff"} Oct 31 01:00:04.925501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.925483 5011 scope.go:122] "RemoveContainer" containerID="1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff" Oct 31 01:00:04.925733 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.925712 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh_odh-ai-gateway-infra(f97a90a6-88a9-48fa-9c79-624fd9bd2b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" Oct 31 01:00:04.927706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.927677 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerID="329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173" exitCode=60 Oct 31 01:00:04.927817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.927740 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" event={"ID":"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90","Type":"ContainerDied","Data":"329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173"} Oct 31 01:00:04.927950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.927935 5011 scope.go:122] "RemoveContainer" containerID="329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173" Oct 31 01:00:04.928151 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:04.928129 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm_odh-ai-gateway-infra(0a4be7a7-e9a7-441f-ae8a-bfbd86daec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" Oct 31 01:00:04.938006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.937885 5011 scope.go:122] "RemoveContainer" containerID="a9bc70d7e479f168271236522d790a69ee1451bdbb587336839aba13895e3b7c" Oct 31 01:00:04.955881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.955857 5011 scope.go:122] "RemoveContainer" containerID="e25f596d9835acec90de65152235e784a71a8cfbcd9ecbe88cce7c66858a86f4" Oct 31 01:00:04.970995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.970971 5011 scope.go:122] "RemoveContainer" containerID="b6057c920351d8cbd1fab52f6c0dda9ef35ced19323a2620275d49c225790f33" Oct 31 01:00:04.989445 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:04.989423 5011 scope.go:122] "RemoveContainer" containerID="5745adaaee5939919bac6eadbeebdc7f7a99bf59770d2943789631f9e72c62c3" Oct 31 01:00:05.004957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.004936 5011 scope.go:122] "RemoveContainer" containerID="78f4a596aeed10085be401fc9788798579732870fc34b6dfae083dc0470001e3" Oct 31 01:00:05.020938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.020916 5011 scope.go:122] "RemoveContainer" containerID="8b4def47f5568b816e6e97cf42949bf0282f035bef2ba697f5f198b22a4a808e" Oct 31 01:00:05.037382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.037250 5011 scope.go:122] "RemoveContainer" containerID="fc18081c0000e60d0fc02efa29cc971313d409c41f90795dfc2db75fa61adbf0" Oct 31 01:00:05.938301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.938268 5011 scope.go:122] "RemoveContainer" containerID="611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3" Oct 31 01:00:05.938776 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.938654 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw_odh-ai-gateway-infra(45849c90-dd35-4665-8f92-df1e3a20aed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" Oct 31 01:00:05.940227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.940191 5011 scope.go:122] "RemoveContainer" containerID="1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448" Oct 31 01:00:05.940585 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.940557 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg_odh-ai-gateway-infra(93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" Oct 31 01:00:05.942550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.942528 5011 scope.go:122] "RemoveContainer" containerID="439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e" Oct 31 01:00:05.942895 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.942867 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l_odh-ai-gateway-infra(92b95a10-5934-4538-806a-a932441beca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" podUID="92b95a10-5934-4538-806a-a932441beca7" Oct 31 01:00:05.944382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.944349 5011 scope.go:122] "RemoveContainer" containerID="b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f" Oct 31 01:00:05.944663 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.944638 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f_odh-ai-gateway-infra(cad06da2-626f-4eea-97b1-1b47431c255c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" Oct 31 01:00:05.946495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.946474 5011 scope.go:122] "RemoveContainer" containerID="3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8" Oct 31 01:00:05.946779 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.946754 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc_odh-ai-gateway-infra(06b53a2a-bd43-4c27-8815-5f47bad4a011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" Oct 31 01:00:05.948380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.948352 5011 scope.go:122] "RemoveContainer" containerID="7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee" Oct 31 01:00:05.948645 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.948621 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96_odh-ai-gateway-infra(a641442d-72f6-4cc7-a9be-01585637a854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" podUID="a641442d-72f6-4cc7-a9be-01585637a854" Oct 31 01:00:05.950187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.950165 5011 scope.go:122] "RemoveContainer" containerID="1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff" Oct 31 01:00:05.950448 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.950411 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh_odh-ai-gateway-infra(f97a90a6-88a9-48fa-9c79-624fd9bd2b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" Oct 31 01:00:05.951886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.951863 5011 scope.go:122] "RemoveContainer" containerID="329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173" Oct 31 01:00:05.952143 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.952118 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm_odh-ai-gateway-infra(0a4be7a7-e9a7-441f-ae8a-bfbd86daec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" Oct 31 01:00:05.954465 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:05.954445 5011 scope.go:122] "RemoveContainer" containerID="5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62" Oct 31 01:00:05.954726 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:05.954705 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr_odh-ai-gateway-infra(eea47c08-65fa-4182-951e-1e2e3295ce15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" Oct 31 01:00:07.058239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:07.058187 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:00:07.070971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:07.070941 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:00:08.002278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:08.002189 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:00:09.500977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.500940 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:00:09.506886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.506867 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.535995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.534842 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-762753-openshift-default-dockercfg-2crgz\"" Oct 31 01:00:09.542655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.542623 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:00:09.562597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.562751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562653 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.562812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562784 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.562866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562839 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.562928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562880 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.562987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.562956 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nkt\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.563081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.563062 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.563136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.563097 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.563193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.563132 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665001 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665099 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665166 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nkt\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665298 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665465 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665479 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665610 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665732 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.665775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.665760 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.666368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.666344 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.667769 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.667736 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.667862 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.667768 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.688736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.688707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.688832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.688822 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nkt\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:09.821467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:09.821433 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:10.176531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:10.176491 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:00:10.177868 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:00:10.177844 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a83023_2dad_4d0d_a728_77e377e1afb9.slice/crio-7783209aac66d301ab96012b0aa3d0f3aaf611fa68f481d4fae6967f3bfb214a WatchSource:0}: Error finding container 7783209aac66d301ab96012b0aa3d0f3aaf611fa68f481d4fae6967f3bfb214a: Status 404 returned error can't find the container with id 7783209aac66d301ab96012b0aa3d0f3aaf611fa68f481d4fae6967f3bfb214a Oct 31 01:00:10.180666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:10.180635 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:00:10.180749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:10.180697 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:00:10.180749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:10.180724 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:00:11.005858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.005814 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" event={"ID":"d1a83023-2dad-4d0d-a728-77e377e1afb9","Type":"ContainerStarted","Data":"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8"} Oct 31 01:00:11.006304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.005880 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" event={"ID":"d1a83023-2dad-4d0d-a728-77e377e1afb9","Type":"ContainerStarted","Data":"7783209aac66d301ab96012b0aa3d0f3aaf611fa68f481d4fae6967f3bfb214a"} Oct 31 01:00:11.074381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.074323 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podStartSLOduration=2.074307687 podStartE2EDuration="2.074307687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:11.071446305 +0000 UTC m=+2479.520533943" watchObservedRunningTime="2026-10-31 01:00:11.074307687 +0000 UTC m=+2479.523395392" Oct 31 01:00:11.825111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.822584 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:11.831230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.830307 5011 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:00:11.831230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:11.830366 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 31 01:00:12.829228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:12.825556 5011 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:00:12.829228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:12.825619 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 31 01:00:13.149452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.149363 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd"] Oct 31 01:00:13.155959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.155931 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.162274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.162244 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd"] Oct 31 01:00:13.172187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.172165 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:13.257230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.255352 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.257230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.255508 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9gs\" (UniqueName: \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.257230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.255595 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.371229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.370994 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.371229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.371145 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9gs\" (UniqueName: \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.371520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.371242 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.375228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.372248 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.392662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.391893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.405702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.405620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9gs\" (UniqueName: \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\") pod \"maas-api-revoke-sub-3af490a1-nc9kd\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.475950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.475549 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:13.852549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.849612 5011 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:00:13.852549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.849680 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 31 01:00:13.858254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:13.855828 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd"] Oct 31 01:00:14.074093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:14.073172 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" event={"ID":"ad99619e-203e-41b8-a029-6b434053aae1","Type":"ContainerStarted","Data":"b6576e47a53a2bc1682813e4115d07af94bb2bbc4947d815358126a685fa6ff0"} Oct 31 01:00:14.823228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:14.822626 5011 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:00:14.823228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:14.822687 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 31 01:00:15.083636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.083605 5011 generic.go:356] "Generic (PLEG): container finished" podID="ad99619e-203e-41b8-a029-6b434053aae1" containerID="7a7aca03d153343167e75616f6fbb1dc429a860bcd518af78916be0b2af18e7b" exitCode=0 Oct 31 01:00:15.083979 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.083843 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" event={"ID":"ad99619e-203e-41b8-a029-6b434053aae1","Type":"ContainerDied","Data":"7a7aca03d153343167e75616f6fbb1dc429a860bcd518af78916be0b2af18e7b"} Oct 31 01:00:15.676103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.676072 5011 scope.go:122] "RemoveContainer" containerID="5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62" Oct 31 01:00:15.676326 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.676262 5011 scope.go:122] "RemoveContainer" containerID="b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f" Oct 31 01:00:15.676460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.676439 5011 scope.go:122] "RemoveContainer" containerID="1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448" Oct 31 01:00:15.843907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.840611 5011 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:00:15.843907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:15.840672 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 31 01:00:16.372629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.372600 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:16.426006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.425971 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\") pod \"ad99619e-203e-41b8-a029-6b434053aae1\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " Oct 31 01:00:16.426277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.426018 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\") pod \"ad99619e-203e-41b8-a029-6b434053aae1\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " Oct 31 01:00:16.426384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.426284 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\" (UniqueName: \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\") pod \"ad99619e-203e-41b8-a029-6b434053aae1\" (UID: \"ad99619e-203e-41b8-a029-6b434053aae1\") " Oct 31 01:00:16.435225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.428747 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca" pod "ad99619e-203e-41b8-a029-6b434053aae1" (UID: "ad99619e-203e-41b8-a029-6b434053aae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:16.435225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.429436 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs" pod "ad99619e-203e-41b8-a029-6b434053aae1" (UID: "ad99619e-203e-41b8-a029-6b434053aae1"). InnerVolumeSpecName "kube-api-access-bq9gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:16.435225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.431325 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca" pod "ad99619e-203e-41b8-a029-6b434053aae1" (UID: "ad99619e-203e-41b8-a029-6b434053aae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:16.528257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.528151 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9gs\" (UniqueName: \"kubernetes.io/projected/ad99619e-203e-41b8-a029-6b434053aae1-kube-api-access-bq9gs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:16.528257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.528183 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:16.528257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.528194 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad99619e-203e-41b8-a029-6b434053aae1-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:16.676368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.676328 5011 scope.go:122] "RemoveContainer" containerID="1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff" Oct 31 01:00:16.676700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.676673 5011 scope.go:122] "RemoveContainer" containerID="7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee" Oct 31 01:00:16.827598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.827570 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:16.828336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.828244 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:16.828918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:16.828899 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:00:17.107411 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.106606 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" event={"ID":"ad99619e-203e-41b8-a029-6b434053aae1","Type":"ContainerDied","Data":"b6576e47a53a2bc1682813e4115d07af94bb2bbc4947d815358126a685fa6ff0"} Oct 31 01:00:17.107411 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.106667 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6576e47a53a2bc1682813e4115d07af94bb2bbc4947d815358126a685fa6ff0" Oct 31 01:00:17.107411 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.106752 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd" Oct 31 01:00:17.113153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.113123 5011 generic.go:356] "Generic (PLEG): container finished" podID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerID="5e73a283e5508f1ea285142f8ebc31a10a1174e2e62fc493860059d6fc86f3b7" exitCode=60 Oct 31 01:00:17.113344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.113236 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerDied","Data":"5e73a283e5508f1ea285142f8ebc31a10a1174e2e62fc493860059d6fc86f3b7"} Oct 31 01:00:17.113344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.113284 5011 scope.go:122] "RemoveContainer" containerID="5d89982cd5fd9173ae65fb9db1da6ad2f8bfe441d8e367a3cb52f506ad5cff62" Oct 31 01:00:17.113811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.113796 5011 scope.go:122] "RemoveContainer" containerID="5e73a283e5508f1ea285142f8ebc31a10a1174e2e62fc493860059d6fc86f3b7" Oct 31 01:00:17.114170 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:17.114147 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr_odh-ai-gateway-infra(eea47c08-65fa-4182-951e-1e2e3295ce15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" Oct 31 01:00:17.117696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.117667 5011 generic.go:356] "Generic (PLEG): container finished" podID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerID="ae778dec1fd22224184ef6fb979dedaa38e8c6517ebc4348803bc0688ae81744" exitCode=60 Oct 31 01:00:17.117903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.117879 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerDied","Data":"ae778dec1fd22224184ef6fb979dedaa38e8c6517ebc4348803bc0688ae81744"} Oct 31 01:00:17.118685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.118332 5011 scope.go:122] "RemoveContainer" containerID="ae778dec1fd22224184ef6fb979dedaa38e8c6517ebc4348803bc0688ae81744" Oct 31 01:00:17.118685 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:17.118646 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg_odh-ai-gateway-infra(93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" Oct 31 01:00:17.127608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.123828 5011 generic.go:356] "Generic (PLEG): container finished" podID="cad06da2-626f-4eea-97b1-1b47431c255c" containerID="0192b513e9a1867463574e6af8c3e8f7742050ce078abb5eafeae126ad87b6da" exitCode=60 Oct 31 01:00:17.127608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.123860 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerDied","Data":"0192b513e9a1867463574e6af8c3e8f7742050ce078abb5eafeae126ad87b6da"} Oct 31 01:00:17.127608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.124431 5011 scope.go:122] "RemoveContainer" containerID="0192b513e9a1867463574e6af8c3e8f7742050ce078abb5eafeae126ad87b6da" Oct 31 01:00:17.127608 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:17.124723 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f_odh-ai-gateway-infra(cad06da2-626f-4eea-97b1-1b47431c255c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" Oct 31 01:00:17.155450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.154318 5011 scope.go:122] "RemoveContainer" containerID="1ad2e51593a9e761810ecb3a85e6502dc5aedf74f632b3de6cd2cf7a2e692448" Oct 31 01:00:17.172822 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.172800 5011 scope.go:122] "RemoveContainer" containerID="b09884ca5287cd2085f0bfcc629b20d7a71ff87499197a4550fa6231193f1e8f" Oct 31 01:00:17.675907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:17.675878 5011 scope.go:122] "RemoveContainer" containerID="611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3" Oct 31 01:00:18.142475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.142440 5011 generic.go:356] "Generic (PLEG): container finished" podID="a641442d-72f6-4cc7-a9be-01585637a854" containerID="b9f90b624289f39833e7d6bfd41927e53bfb29b7ebcdca6bfaf9a93c9d475e27" exitCode=60 Oct 31 01:00:18.142651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.142507 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerDied","Data":"b9f90b624289f39833e7d6bfd41927e53bfb29b7ebcdca6bfaf9a93c9d475e27"} Oct 31 01:00:18.142651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.142574 5011 scope.go:122] "RemoveContainer" containerID="7b986760c7a9711c5951c898af849fcc4f5acf9dd51b38474a96edc528168bee" Oct 31 01:00:18.143003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.142982 5011 scope.go:122] "RemoveContainer" containerID="b9f90b624289f39833e7d6bfd41927e53bfb29b7ebcdca6bfaf9a93c9d475e27" Oct 31 01:00:18.143401 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:18.143372 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96_odh-ai-gateway-infra(a641442d-72f6-4cc7-a9be-01585637a854)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" podUID="a641442d-72f6-4cc7-a9be-01585637a854" Oct 31 01:00:18.145004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.144983 5011 generic.go:356] "Generic (PLEG): container finished" podID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerID="deeb084ffa53cdd67e39a66b61040b9f0b12d7c8bfbdc5fcd377d339b52dded0" exitCode=60 Oct 31 01:00:18.145095 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.145051 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" event={"ID":"f97a90a6-88a9-48fa-9c79-624fd9bd2b28","Type":"ContainerDied","Data":"deeb084ffa53cdd67e39a66b61040b9f0b12d7c8bfbdc5fcd377d339b52dded0"} Oct 31 01:00:18.145521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.145505 5011 scope.go:122] "RemoveContainer" containerID="deeb084ffa53cdd67e39a66b61040b9f0b12d7c8bfbdc5fcd377d339b52dded0" Oct 31 01:00:18.145899 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:18.145875 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh_odh-ai-gateway-infra(f97a90a6-88a9-48fa-9c79-624fd9bd2b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" Oct 31 01:00:18.156303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.156285 5011 scope.go:122] "RemoveContainer" containerID="1cdf59666c4e2cea6b9a00bbb7924f552929e967a2e1da62056afad002e5a5ff" Oct 31 01:00:18.183852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.183794 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:18.232784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.232748 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:18.290134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.290101 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:18.452830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.452795 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:18.534766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.534733 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:18.612610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.611858 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\" (UniqueName: \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\") pod \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " Oct 31 01:00:18.612610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.611941 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\" (UniqueName: \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\") pod \"eea47c08-65fa-4182-951e-1e2e3295ce15\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " Oct 31 01:00:18.612610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.612057 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\") pod \"eea47c08-65fa-4182-951e-1e2e3295ce15\" (UID: \"eea47c08-65fa-4182-951e-1e2e3295ce15\") " Oct 31 01:00:18.612610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.612141 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\") pod \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\" (UID: \"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7\") " Oct 31 01:00:18.615125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.615075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca" pod "93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" (UID: "93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.615125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.615094 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca" pod "eea47c08-65fa-4182-951e-1e2e3295ce15" (UID: "eea47c08-65fa-4182-951e-1e2e3295ce15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.618322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.618292 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf" pod "93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" (UID: "93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7"). InnerVolumeSpecName "kube-api-access-fkmlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.618789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.618736 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw" pod "eea47c08-65fa-4182-951e-1e2e3295ce15" (UID: "eea47c08-65fa-4182-951e-1e2e3295ce15"). InnerVolumeSpecName "kube-api-access-l4hfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.650653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.650625 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:18.675779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.675746 5011 scope.go:122] "RemoveContainer" containerID="439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e" Oct 31 01:00:18.675976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.675916 5011 scope.go:122] "RemoveContainer" containerID="329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173" Oct 31 01:00:18.714618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.714592 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmlf\" (UniqueName: \"kubernetes.io/projected/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-kube-api-access-fkmlf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:18.714618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.714619 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hfw\" (UniqueName: \"kubernetes.io/projected/eea47c08-65fa-4182-951e-1e2e3295ce15-kube-api-access-l4hfw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:18.714839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.714631 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea47c08-65fa-4182-951e-1e2e3295ce15-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:18.714839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.714640 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:18.815895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.815864 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\") pod \"cad06da2-626f-4eea-97b1-1b47431c255c\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " Oct 31 01:00:18.816126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.816109 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\" (UniqueName: \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\") pod \"cad06da2-626f-4eea-97b1-1b47431c255c\" (UID: \"cad06da2-626f-4eea-97b1-1b47431c255c\") " Oct 31 01:00:18.816255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.816199 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca" pod "cad06da2-626f-4eea-97b1-1b47431c255c" (UID: "cad06da2-626f-4eea-97b1-1b47431c255c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.817069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.817043 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad06da2-626f-4eea-97b1-1b47431c255c-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:18.818788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.818751 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc" pod "cad06da2-626f-4eea-97b1-1b47431c255c" (UID: "cad06da2-626f-4eea-97b1-1b47431c255c"). InnerVolumeSpecName "kube-api-access-sn9mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.918585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:18.918542 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9mc\" (UniqueName: \"kubernetes.io/projected/cad06da2-626f-4eea-97b1-1b47431c255c-kube-api-access-sn9mc\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.158167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.158131 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" event={"ID":"93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7","Type":"ContainerDied","Data":"bfcb2eb7cebdc55b3b5e32c0bc656e317cc66cb70d7e2e5b2f9eb5b559056235"} Oct 31 01:00:19.158467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.158439 5011 scope.go:122] "RemoveContainer" containerID="ae778dec1fd22224184ef6fb979dedaa38e8c6517ebc4348803bc0688ae81744" Oct 31 01:00:19.158632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.158303 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg" Oct 31 01:00:19.165865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.165840 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" event={"ID":"cad06da2-626f-4eea-97b1-1b47431c255c","Type":"ContainerDied","Data":"fbf7b1415c704caa53a872c7ea73b2bd3adfcfc973e39dad3292dcb6c5bf00b8"} Oct 31 01:00:19.165964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.165945 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f" Oct 31 01:00:19.196260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.196178 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" event={"ID":"eea47c08-65fa-4182-951e-1e2e3295ce15","Type":"ContainerDied","Data":"7ebcc9bc92700038b952e963b7b48822b7ce33917bdc5847b1b419708d297e90"} Oct 31 01:00:19.196397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.196290 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr" Oct 31 01:00:19.199884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.199850 5011 generic.go:356] "Generic (PLEG): container finished" podID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerID="a8b75054b82be4c327cc460f1d986c12b111ff5cd367f60ed53c0085ab900089" exitCode=60 Oct 31 01:00:19.200029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.199906 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerDied","Data":"a8b75054b82be4c327cc460f1d986c12b111ff5cd367f60ed53c0085ab900089"} Oct 31 01:00:19.201311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.200452 5011 scope.go:122] "RemoveContainer" containerID="a8b75054b82be4c327cc460f1d986c12b111ff5cd367f60ed53c0085ab900089" Oct 31 01:00:19.201311 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:19.200848 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw_odh-ai-gateway-infra(45849c90-dd35-4665-8f92-df1e3a20aed0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" Oct 31 01:00:19.239571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.238992 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:19.241404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.241384 5011 scope.go:122] "RemoveContainer" containerID="0192b513e9a1867463574e6af8c3e8f7742050ce078abb5eafeae126ad87b6da" Oct 31 01:00:19.260575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.260550 5011 scope.go:122] "RemoveContainer" containerID="5e73a283e5508f1ea285142f8ebc31a10a1174e2e62fc493860059d6fc86f3b7" Oct 31 01:00:19.290498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.290472 5011 scope.go:122] "RemoveContainer" containerID="611a90dbdaebca3643841780dabfce87f0f009acadc1247cbe282ec6d789cea3" Oct 31 01:00:19.294926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.294900 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:19.323552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.323525 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:19.336376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.336340 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890140-v9nmg"] Oct 31 01:00:19.370487 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.370455 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:19.375136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.375103 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890140-m2k6f"] Oct 31 01:00:19.458230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.458111 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:19.463278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.463251 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890140-28pdr"] Oct 31 01:00:19.463450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.463432 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:19.508920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.508898 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:19.638726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.638694 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\") pod \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " Oct 31 01:00:19.638924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.638754 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\") pod \"a641442d-72f6-4cc7-a9be-01585637a854\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " Oct 31 01:00:19.638924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.638798 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\" (UniqueName: \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\") pod \"a641442d-72f6-4cc7-a9be-01585637a854\" (UID: \"a641442d-72f6-4cc7-a9be-01585637a854\") " Oct 31 01:00:19.638924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.638878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\" (UniqueName: \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\") pod \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\" (UID: \"f97a90a6-88a9-48fa-9c79-624fd9bd2b28\") " Oct 31 01:00:19.639088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.639068 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca" pod "f97a90a6-88a9-48fa-9c79-624fd9bd2b28" (UID: "f97a90a6-88a9-48fa-9c79-624fd9bd2b28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.639176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.639152 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca" pod "a641442d-72f6-4cc7-a9be-01585637a854" (UID: "a641442d-72f6-4cc7-a9be-01585637a854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.639738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.639714 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.639738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.639738 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a641442d-72f6-4cc7-a9be-01585637a854-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.641025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.641002 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt" pod "a641442d-72f6-4cc7-a9be-01585637a854" (UID: "a641442d-72f6-4cc7-a9be-01585637a854"). InnerVolumeSpecName "kube-api-access-xgnzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.641112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.641030 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm" pod "f97a90a6-88a9-48fa-9c79-624fd9bd2b28" (UID: "f97a90a6-88a9-48fa-9c79-624fd9bd2b28"). InnerVolumeSpecName "kube-api-access-gvmfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.675580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.675550 5011 scope.go:122] "RemoveContainer" containerID="3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8" Oct 31 01:00:19.685029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.685006 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" path="/var/lib/kubelet/pods/93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7/volumes" Oct 31 01:00:19.685466 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.685451 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" path="/var/lib/kubelet/pods/cad06da2-626f-4eea-97b1-1b47431c255c/volumes" Oct 31 01:00:19.685870 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.685857 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" path="/var/lib/kubelet/pods/eea47c08-65fa-4182-951e-1e2e3295ce15/volumes" Oct 31 01:00:19.742084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.742052 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmfm\" (UniqueName: \"kubernetes.io/projected/f97a90a6-88a9-48fa-9c79-624fd9bd2b28-kube-api-access-gvmfm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.742084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:19.742086 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnzt\" (UniqueName: \"kubernetes.io/projected/a641442d-72f6-4cc7-a9be-01585637a854-kube-api-access-xgnzt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:20.214009 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.213977 5011 generic.go:356] "Generic (PLEG): container finished" podID="92b95a10-5934-4538-806a-a932441beca7" containerID="3bd42eb7d262dd838485ab2ca25021f7f80d1d86cab5d97e3d15a1c4a4e2ff2c" exitCode=60 Oct 31 01:00:20.214255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.214049 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerDied","Data":"3bd42eb7d262dd838485ab2ca25021f7f80d1d86cab5d97e3d15a1c4a4e2ff2c"} Oct 31 01:00:20.214255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.214113 5011 scope.go:122] "RemoveContainer" containerID="439a59b3bc0a40f402c444f9006f84baf62f417d83df955380f051e0631e5b6e" Oct 31 01:00:20.214626 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.214598 5011 scope.go:122] "RemoveContainer" containerID="3bd42eb7d262dd838485ab2ca25021f7f80d1d86cab5d97e3d15a1c4a4e2ff2c" Oct 31 01:00:20.215042 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:20.215018 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l_odh-ai-gateway-infra(92b95a10-5934-4538-806a-a932441beca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" podUID="92b95a10-5934-4538-806a-a932441beca7" Oct 31 01:00:20.217463 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.217440 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerStarted","Data":"dc1f9175c6d9d4973eb6dea184d579eb5a9a3f934fc3bcc7e761f3c7dd3afc4c"} Oct 31 01:00:20.219133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.219114 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" Oct 31 01:00:20.219133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.219124 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96" event={"ID":"a641442d-72f6-4cc7-a9be-01585637a854","Type":"ContainerDied","Data":"b662d2f03b41b8bf491c1c5d11943cbec4684e088e7b4db76e93ca8984a7b890"} Oct 31 01:00:20.220651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.220627 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" event={"ID":"f97a90a6-88a9-48fa-9c79-624fd9bd2b28","Type":"ContainerDied","Data":"d4cfba7b97fb60abdc4d4326a6ce36686dbef0cce73e7b14ea1706f0612eccf7"} Oct 31 01:00:20.220773 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.220632 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh" Oct 31 01:00:20.222763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.222742 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerID="e5e432c339e94354e0a9cff936aeb6ff8095236cef9818a90f350ba1840df2ed" exitCode=60 Oct 31 01:00:20.222841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.222794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" event={"ID":"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90","Type":"ContainerDied","Data":"e5e432c339e94354e0a9cff936aeb6ff8095236cef9818a90f350ba1840df2ed"} Oct 31 01:00:20.223154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.223133 5011 scope.go:122] "RemoveContainer" containerID="e5e432c339e94354e0a9cff936aeb6ff8095236cef9818a90f350ba1840df2ed" Oct 31 01:00:20.223459 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:20.223439 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm_odh-ai-gateway-infra(0a4be7a7-e9a7-441f-ae8a-bfbd86daec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" Oct 31 01:00:20.235241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.235169 5011 scope.go:122] "RemoveContainer" containerID="b9f90b624289f39833e7d6bfd41927e53bfb29b7ebcdca6bfaf9a93c9d475e27" Oct 31 01:00:20.246627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.246609 5011 scope.go:122] "RemoveContainer" containerID="deeb084ffa53cdd67e39a66b61040b9f0b12d7c8bfbdc5fcd377d339b52dded0" Oct 31 01:00:20.256424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.256399 5011 scope.go:122] "RemoveContainer" containerID="329af0c9ed35e91783f6c52f4c02cb69b9637464cc87f5e172ad7c2148e0e173" Oct 31 01:00:20.281121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.281099 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:20.347105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.347044 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:20.353770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.353299 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890140-s7d96"] Oct 31 01:00:20.389319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.389282 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:20.394125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.394097 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6d97d-29890140-5z9hh"] Oct 31 01:00:20.488546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.488481 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:20.659906 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.659874 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\" (UniqueName: \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\") pod \"45849c90-dd35-4665-8f92-df1e3a20aed0\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " Oct 31 01:00:20.660836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.659924 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\") pod \"45849c90-dd35-4665-8f92-df1e3a20aed0\" (UID: \"45849c90-dd35-4665-8f92-df1e3a20aed0\") " Oct 31 01:00:20.660836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.660515 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca" pod "45849c90-dd35-4665-8f92-df1e3a20aed0" (UID: "45849c90-dd35-4665-8f92-df1e3a20aed0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.664386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.664357 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw" pod "45849c90-dd35-4665-8f92-df1e3a20aed0" (UID: "45849c90-dd35-4665-8f92-df1e3a20aed0"). InnerVolumeSpecName "kube-api-access-cr5xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.763276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.763176 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr5xw\" (UniqueName: \"kubernetes.io/projected/45849c90-dd35-4665-8f92-df1e3a20aed0-kube-api-access-cr5xw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:20.763276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:20.763227 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45849c90-dd35-4665-8f92-df1e3a20aed0-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.249939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.249893 5011 generic.go:356] "Generic (PLEG): container finished" podID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerID="dc1f9175c6d9d4973eb6dea184d579eb5a9a3f934fc3bcc7e761f3c7dd3afc4c" exitCode=60 Oct 31 01:00:21.250114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.249971 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerDied","Data":"dc1f9175c6d9d4973eb6dea184d579eb5a9a3f934fc3bcc7e761f3c7dd3afc4c"} Oct 31 01:00:21.250114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.250082 5011 scope.go:122] "RemoveContainer" containerID="3eeb9af6143a7bfa9914e5594a079823374953ebd23b25ddd5e7beabca3789e8" Oct 31 01:00:21.250514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.250498 5011 scope.go:122] "RemoveContainer" containerID="dc1f9175c6d9d4973eb6dea184d579eb5a9a3f934fc3bcc7e761f3c7dd3afc4c" Oct 31 01:00:21.250842 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:21.250823 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc_odh-ai-gateway-infra(06b53a2a-bd43-4c27-8815-5f47bad4a011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" Oct 31 01:00:21.256437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.256415 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" Oct 31 01:00:21.256437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.256438 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw" event={"ID":"45849c90-dd35-4665-8f92-df1e3a20aed0","Type":"ContainerDied","Data":"e3ae6d4e674d2e1fec91d293e72b6b78d7183db10b72451b124706922a44a462"} Oct 31 01:00:21.267356 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.267278 5011 scope.go:122] "RemoveContainer" containerID="a8b75054b82be4c327cc460f1d986c12b111ff5cd367f60ed53c0085ab900089" Oct 31 01:00:21.272032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.272010 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:21.317101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.317072 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:21.391443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.389074 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:21.395069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.395036 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890140-dmjkw"] Oct 31 01:00:21.441521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.441489 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:21.506006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.505947 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:21.575412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.575374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\" (UniqueName: \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\") pod \"92b95a10-5934-4538-806a-a932441beca7\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " Oct 31 01:00:21.575585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.575563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\") pod \"92b95a10-5934-4538-806a-a932441beca7\" (UID: \"92b95a10-5934-4538-806a-a932441beca7\") " Oct 31 01:00:21.576387 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.576335 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca" pod "92b95a10-5934-4538-806a-a932441beca7" (UID: "92b95a10-5934-4538-806a-a932441beca7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.589478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.578263 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm" pod "92b95a10-5934-4538-806a-a932441beca7" (UID: "92b95a10-5934-4538-806a-a932441beca7"). InnerVolumeSpecName "kube-api-access-db9sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.633709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.633672 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 01:00:21.633901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.633879 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" containerID="cri-o://1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207" gracePeriod=30 Oct 31 01:00:21.637346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.637322 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:21.676622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.676590 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\") pod \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " Oct 31 01:00:21.676787 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.676743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\" (UniqueName: \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\") pod \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\" (UID: \"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90\") " Oct 31 01:00:21.677262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.676928 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca" pod "0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" (UID: "0a4be7a7-e9a7-441f-ae8a-bfbd86daec90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.677262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.677240 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9sm\" (UniqueName: \"kubernetes.io/projected/92b95a10-5934-4538-806a-a932441beca7-kube-api-access-db9sm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.677262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.677264 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b95a10-5934-4538-806a-a932441beca7-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.677526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.677273 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.679615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.679586 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762" pod "0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" (UID: "0a4be7a7-e9a7-441f-ae8a-bfbd86daec90"). InnerVolumeSpecName "kube-api-access-7m762". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.691459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.691427 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" path="/var/lib/kubelet/pods/45849c90-dd35-4665-8f92-df1e3a20aed0/volumes" Oct 31 01:00:21.692075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.692053 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a641442d-72f6-4cc7-a9be-01585637a854" path="/var/lib/kubelet/pods/a641442d-72f6-4cc7-a9be-01585637a854/volumes" Oct 31 01:00:21.692685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.692664 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" path="/var/lib/kubelet/pods/f97a90a6-88a9-48fa-9c79-624fd9bd2b28/volumes" Oct 31 01:00:21.779636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:21.779320 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m762\" (UniqueName: \"kubernetes.io/projected/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90-kube-api-access-7m762\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.274890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.274850 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" Oct 31 01:00:22.274890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.274880 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l" event={"ID":"92b95a10-5934-4538-806a-a932441beca7","Type":"ContainerDied","Data":"48f34b9eee0289912e7f1822fffe641908f8215e33e3dabcb1acb7d03498a6d0"} Oct 31 01:00:22.275308 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.274952 5011 scope.go:122] "RemoveContainer" containerID="3bd42eb7d262dd838485ab2ca25021f7f80d1d86cab5d97e3d15a1c4a4e2ff2c" Oct 31 01:00:22.280086 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.279502 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" Oct 31 01:00:22.280086 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.279522 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm" event={"ID":"0a4be7a7-e9a7-441f-ae8a-bfbd86daec90","Type":"ContainerDied","Data":"d57b20105cdeecb44b9c8f3f8f5f5b585c4cad9ed704adaa3ff85dc77dcb074e"} Oct 31 01:00:22.296535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.296460 5011 scope.go:122] "RemoveContainer" containerID="e5e432c339e94354e0a9cff936aeb6ff8095236cef9818a90f350ba1840df2ed" Oct 31 01:00:22.396148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.396103 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:22.413867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.413834 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890140-sc69l"] Oct 31 01:00:22.448945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.448905 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:22.458399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.458367 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd1dfc-29890140-68qzm"] Oct 31 01:00:22.532273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.532195 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:22.594616 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.594573 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\") pod \"06b53a2a-bd43-4c27-8815-5f47bad4a011\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " Oct 31 01:00:22.594818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.594692 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\" (UniqueName: \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\") pod \"06b53a2a-bd43-4c27-8815-5f47bad4a011\" (UID: \"06b53a2a-bd43-4c27-8815-5f47bad4a011\") " Oct 31 01:00:22.596033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.596005 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca" pod "06b53a2a-bd43-4c27-8815-5f47bad4a011" (UID: "06b53a2a-bd43-4c27-8815-5f47bad4a011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:22.598740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.598708 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7" pod "06b53a2a-bd43-4c27-8815-5f47bad4a011" (UID: "06b53a2a-bd43-4c27-8815-5f47bad4a011"). InnerVolumeSpecName "kube-api-access-l4hk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:22.697368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.697334 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b53a2a-bd43-4c27-8815-5f47bad4a011-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.697368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:22.697369 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hk7\" (UniqueName: \"kubernetes.io/projected/06b53a2a-bd43-4c27-8815-5f47bad4a011-kube-api-access-l4hk7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:23.297956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.297915 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" event={"ID":"06b53a2a-bd43-4c27-8815-5f47bad4a011","Type":"ContainerDied","Data":"ec1144731118cab1f270a4d45809d7e833abff35f4a8a9720fad345d02dd6e3b"} Oct 31 01:00:23.298539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.297989 5011 scope.go:122] "RemoveContainer" containerID="dc1f9175c6d9d4973eb6dea184d579eb5a9a3f934fc3bcc7e761f3c7dd3afc4c" Oct 31 01:00:23.298539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.298084 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc" Oct 31 01:00:23.545738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.545705 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:23.556414 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.556334 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4e69-29890140-b7mtc"] Oct 31 01:00:23.687143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.687103 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" path="/var/lib/kubelet/pods/06b53a2a-bd43-4c27-8815-5f47bad4a011/volumes" Oct 31 01:00:23.687855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.687827 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" path="/var/lib/kubelet/pods/0a4be7a7-e9a7-441f-ae8a-bfbd86daec90/volumes" Oct 31 01:00:23.688520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:23.688500 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b95a10-5934-4538-806a-a932441beca7" path="/var/lib/kubelet/pods/92b95a10-5934-4538-806a-a932441beca7/volumes" Oct 31 01:00:26.939145 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:26.939110 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 01:00:27.003543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003506 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003587 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003629 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003721 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003798 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003861 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.003966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.003951 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.004049 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.004000 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.004102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.004055 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\") pod \"53214e19-073f-4c12-928b-51799031a596\" (UID: \"53214e19-073f-4c12-928b-51799031a596\") " Oct 31 01:00:27.005889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.005297 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.005889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.005873 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.006258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.006234 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:27.006582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.006533 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.006846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.006825 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.015578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.015549 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:27.022245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.016461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "kube-api-access-ktt6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.030292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.030258 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.030437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.030415 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy" pod "53214e19-073f-4c12-928b-51799031a596" (UID: "53214e19-073f-4c12-928b-51799031a596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137803 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137837 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53214e19-073f-4c12-928b-51799031a596-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137852 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137864 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53214e19-073f-4c12-928b-51799031a596-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137877 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137890 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137903 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137918 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktt6w\" (UniqueName: \"kubernetes.io/projected/53214e19-073f-4c12-928b-51799031a596-kube-api-access-ktt6w\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.150944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.137931 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53214e19-073f-4c12-928b-51799031a596-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.336897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.336855 5011 generic.go:356] "Generic (PLEG): container finished" podID="53214e19-073f-4c12-928b-51799031a596" containerID="1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207" exitCode=0 Oct 31 01:00:27.337072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.336903 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" Oct 31 01:00:27.337072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.336936 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" event={"ID":"53214e19-073f-4c12-928b-51799031a596","Type":"ContainerDied","Data":"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207"} Oct 31 01:00:27.337072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.337036 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf" event={"ID":"53214e19-073f-4c12-928b-51799031a596","Type":"ContainerDied","Data":"ee9a60bbf99665ffbbddf087ebc2a96140fb7896797a6cab547edfe847182b27"} Oct 31 01:00:27.337072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.337067 5011 scope.go:122] "RemoveContainer" containerID="1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207" Oct 31 01:00:27.350888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.350872 5011 scope.go:122] "RemoveContainer" containerID="1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207" Oct 31 01:00:27.351126 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:27.351108 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207\": container with ID starting with 1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207 not found: ID does not exist" containerID="1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207" Oct 31 01:00:27.351199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.351134 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207"} err="failed to get container status \"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207\": rpc error: code = NotFound desc = could not find container \"1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207\": container with ID starting with 1bf1d2b47a9a80cf714deb84d93b9e462eab17d775c6f20cd54301e9481d3207 not found: ID does not exist" Oct 31 01:00:27.393327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.393291 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 01:00:27.403902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.403882 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-lhgqf"] Oct 31 01:00:27.686427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:27.686350 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53214e19-073f-4c12-928b-51799031a596" path="/var/lib/kubelet/pods/53214e19-073f-4c12-928b-51799031a596/volumes" Oct 31 01:00:31.951228 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:00:31.950418 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:00:45.054055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:45.054015 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:00:53.314264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:53.314226 5011 scope.go:122] "RemoveContainer" containerID="17fc0e04f8563ca07e8ef54e4079834c99c49ac819b7afb071a912d55ca763dc" Oct 31 01:00:53.328829 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:53.328782 5011 scope.go:122] "RemoveContainer" containerID="9d0d7edd40ee01cf8afed8646241344eb4a96863c5e174bfe2b13db76c0b7831" Oct 31 01:00:53.338783 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:53.338763 5011 scope.go:122] "RemoveContainer" containerID="245b868802f3135bdca61a0126fd9b49178a8ec305e6880da66571d657a5cc21" Oct 31 01:00:53.439801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:53.439764 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 01:00:53.440103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:53.440073 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" containerID="cri-o://05fafc6d7770f03b7db397beca6564b8bf4f6184dd364d5432e1c7281746a018" gracePeriod=30 Oct 31 01:00:58.655045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.654999 5011 generic.go:356] "Generic (PLEG): container finished" podID="7ad55bd2-5eae-4763-b439-c254281da697" containerID="05fafc6d7770f03b7db397beca6564b8bf4f6184dd364d5432e1c7281746a018" exitCode=0 Oct 31 01:00:58.655450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.655064 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" event={"ID":"7ad55bd2-5eae-4763-b439-c254281da697","Type":"ContainerDied","Data":"05fafc6d7770f03b7db397beca6564b8bf4f6184dd364d5432e1c7281746a018"} Oct 31 01:00:58.788147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.788119 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 01:00:58.909410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.909362 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.909602 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.909714 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.909909 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910014 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910062 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910225 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910261 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910330 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910389 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\") pod \"7ad55bd2-5eae-4763-b439-c254281da697\" (UID: \"7ad55bd2-5eae-4763-b439-c254281da697\") " Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910471 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910630 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.910900 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.911167 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.911187 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.911223 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.911242 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad55bd2-5eae-4763-b439-c254281da697-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:58.916403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.911522 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:58.917932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.917906 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:58.919686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.919522 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:58.927411 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.927379 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "kube-api-access-mhbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:58.960681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:58.960323 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy" pod "7ad55bd2-5eae-4763-b439-c254281da697" (UID: "7ad55bd2-5eae-4763-b439-c254281da697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:59.012634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.012553 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:59.012634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.012589 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad55bd2-5eae-4763-b439-c254281da697-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:59.012634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.012603 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbs5\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-kube-api-access-mhbs5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:59.012634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.012617 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad55bd2-5eae-4763-b439-c254281da697-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:59.012634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.012631 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad55bd2-5eae-4763-b439-c254281da697-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:00:59.664727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.664694 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" Oct 31 01:00:59.664727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.664715 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2" event={"ID":"7ad55bd2-5eae-4763-b439-c254281da697","Type":"ContainerDied","Data":"8deb0bc828fb3c8801ecf03cc2d1e74e035aa1d9c3053ad0e95a316614ba4a02"} Oct 31 01:00:59.665243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.664783 5011 scope.go:122] "RemoveContainer" containerID="05fafc6d7770f03b7db397beca6564b8bf4f6184dd364d5432e1c7281746a018" Oct 31 01:00:59.731028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.730999 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 01:00:59.743383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:00:59.743356 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d5621-openshift-default-555dc97b75-gcch2"] Oct 31 01:01:01.007253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:01.003324 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 01:01:01.686014 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:01.685982 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad55bd2-5eae-4763-b439-c254281da697" path="/var/lib/kubelet/pods/7ad55bd2-5eae-4763-b439-c254281da697/volumes" Oct 31 01:01:01.949773 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:01:01.949687 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:01:16.045311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.045267 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" podUID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" containerName="main" containerID="cri-o://b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93" gracePeriod=45 Oct 31 01:01:16.403023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.402991 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 01:01:16.537956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.537919 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.538146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538074 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\" (UniqueName: \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.538257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538177 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.538331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538277 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538432 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538466 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\") pod \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\" (UID: \"a066ab6d-e8ed-44bc-addc-b0e5859462d5\") " Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538273 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538495 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.538701 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.539182 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.539223 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.539262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.539237 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.540710 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.540681 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.540829 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.540721 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:16.540933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.540913 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr" pod "a066ab6d-e8ed-44bc-addc-b0e5859462d5" (UID: "a066ab6d-e8ed-44bc-addc-b0e5859462d5"). InnerVolumeSpecName "kube-api-access-6lprr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:16.640789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.640709 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lprr\" (UniqueName: \"kubernetes.io/projected/a066ab6d-e8ed-44bc-addc-b0e5859462d5-kube-api-access-6lprr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.640789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.640746 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a066ab6d-e8ed-44bc-addc-b0e5859462d5-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.640789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.640762 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a066ab6d-e8ed-44bc-addc-b0e5859462d5-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:16.812887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.812855 5011 generic.go:356] "Generic (PLEG): container finished" podID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" containerID="b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93" exitCode=0 Oct 31 01:01:16.813066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.812899 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" Oct 31 01:01:16.813066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.812949 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" event={"ID":"a066ab6d-e8ed-44bc-addc-b0e5859462d5","Type":"ContainerDied","Data":"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93"} Oct 31 01:01:16.813066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.812993 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45" event={"ID":"a066ab6d-e8ed-44bc-addc-b0e5859462d5","Type":"ContainerDied","Data":"4d54d2d2454b5e564fb4566a6cdb2acbb313546e7ef939b4cc1bafdcb9caba5a"} Oct 31 01:01:16.813066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.813015 5011 scope.go:122] "RemoveContainer" containerID="b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93" Oct 31 01:01:16.824727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.824707 5011 scope.go:122] "RemoveContainer" containerID="b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93" Oct 31 01:01:16.825014 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:01:16.824992 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93\": container with ID starting with b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93 not found: ID does not exist" containerID="b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93" Oct 31 01:01:16.825089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.825022 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93"} err="failed to get container status \"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93\": rpc error: code = NotFound desc = could not find container \"b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93\": container with ID starting with b6883c30c14bbf0703832148341fcebd4c929eebf4d3619e63226b214cb25c93 not found: ID does not exist" Oct 31 01:01:16.867834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.867805 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 01:01:16.878697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:16.878672 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-609843/test-model-609843-kserve-649cd84dff-77r45"] Oct 31 01:01:17.684068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:17.684029 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" path="/var/lib/kubelet/pods/a066ab6d-e8ed-44bc-addc-b0e5859462d5/volumes" Oct 31 01:01:31.952403 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:01:31.952365 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:01:40.966895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:40.966759 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 01:01:40.967679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:40.967624 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" podUID="da76ea30-ff43-4aed-ac48-5483bc01983c" containerName="maas-api" containerID="cri-o://f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab" gracePeriod=30 Oct 31 01:01:42.061473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.061389 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 01:01:42.174885 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.174851 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\") pod \"da76ea30-ff43-4aed-ac48-5483bc01983c\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " Oct 31 01:01:42.175415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.175393 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\") pod \"da76ea30-ff43-4aed-ac48-5483bc01983c\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " Oct 31 01:01:42.175577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.175563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\") pod \"da76ea30-ff43-4aed-ac48-5483bc01983c\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " Oct 31 01:01:42.175698 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.175686 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\" (UniqueName: \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\") pod \"da76ea30-ff43-4aed-ac48-5483bc01983c\" (UID: \"da76ea30-ff43-4aed-ac48-5483bc01983c\") " Oct 31 01:01:42.180809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.180780 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca" pod "da76ea30-ff43-4aed-ac48-5483bc01983c" (UID: "da76ea30-ff43-4aed-ac48-5483bc01983c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:42.218135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.215477 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls" pod "da76ea30-ff43-4aed-ac48-5483bc01983c" (UID: "da76ea30-ff43-4aed-ac48-5483bc01983c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:42.218135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.217456 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs" pod "da76ea30-ff43-4aed-ac48-5483bc01983c" (UID: "da76ea30-ff43-4aed-ac48-5483bc01983c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:42.218135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.217565 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb" pod "da76ea30-ff43-4aed-ac48-5483bc01983c" (UID: "da76ea30-ff43-4aed-ac48-5483bc01983c"). InnerVolumeSpecName "kube-api-access-mpgwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:42.219162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.218646 5011 generic.go:356] "Generic (PLEG): container finished" podID="da76ea30-ff43-4aed-ac48-5483bc01983c" containerID="f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab" exitCode=0 Oct 31 01:01:42.219162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.218700 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" event={"ID":"da76ea30-ff43-4aed-ac48-5483bc01983c","Type":"ContainerDied","Data":"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab"} Oct 31 01:01:42.219162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.218748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" event={"ID":"da76ea30-ff43-4aed-ac48-5483bc01983c","Type":"ContainerDied","Data":"92f131ab4cff3eeeb6f54ebd7e42cb8942dee36443354ac7133782458ec2b982"} Oct 31 01:01:42.219162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.218781 5011 scope.go:122] "RemoveContainer" containerID="f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab" Oct 31 01:01:42.219162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.218943 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp" Oct 31 01:01:42.259253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.259229 5011 scope.go:122] "RemoveContainer" containerID="f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab" Oct 31 01:01:42.259625 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:01:42.259597 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab\": container with ID starting with f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab not found: ID does not exist" containerID="f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab" Oct 31 01:01:42.259699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.259639 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab"} err="failed to get container status \"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab\": rpc error: code = NotFound desc = could not find container \"f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab\": container with ID starting with f8827cf43f3f2b435b7725d891309da671e0c2d30c15a0261da166f56ff730ab not found: ID does not exist" Oct 31 01:01:42.277918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.277885 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:42.277918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.277913 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da76ea30-ff43-4aed-ac48-5483bc01983c-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:42.278081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.277927 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da76ea30-ff43-4aed-ac48-5483bc01983c-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:42.278081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.277936 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgwb\" (UniqueName: \"kubernetes.io/projected/da76ea30-ff43-4aed-ac48-5483bc01983c-kube-api-access-mpgwb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:01:42.293074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.293046 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 01:01:42.296920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:42.296896 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6d97d-544557c654-k95wp"] Oct 31 01:01:43.684611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:01:43.684574 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da76ea30-ff43-4aed-ac48-5483bc01983c" path="/var/lib/kubelet/pods/da76ea30-ff43-4aed-ac48-5483bc01983c/volumes" Oct 31 01:02:01.981092 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:01.981050 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:02:06.071170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:06.071136 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 01:02:06.071587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:06.071380 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" containerID="cri-o://ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4" gracePeriod=30 Oct 31 01:02:10.869505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:10.869469 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:10.869897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:10.869533 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:11.393326 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.393298 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 01:02:11.497970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.497865 5011 generic.go:356] "Generic (PLEG): container finished" podID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerID="ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4" exitCode=0 Oct 31 01:02:11.497970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.497910 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" Oct 31 01:02:11.497970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.497950 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" event={"ID":"8a2ed75c-5330-4a56-8ed6-f359274dee68","Type":"ContainerDied","Data":"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4"} Oct 31 01:02:11.498324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.498007 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457" event={"ID":"8a2ed75c-5330-4a56-8ed6-f359274dee68","Type":"ContainerDied","Data":"b523a96f116ec0dbda0bf0398ca80767d0a85b1ceee561bca94a9bdab77afac1"} Oct 31 01:02:11.498324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.498038 5011 scope.go:122] "RemoveContainer" containerID="ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4" Oct 31 01:02:11.511270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.511246 5011 scope.go:122] "RemoveContainer" containerID="ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4" Oct 31 01:02:11.511546 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:11.511518 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4\": container with ID starting with ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4 not found: ID does not exist" containerID="ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4" Oct 31 01:02:11.511631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.511557 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4"} err="failed to get container status \"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4\": rpc error: code = NotFound desc = could not find container \"ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4\": container with ID starting with ab09fc83e63f79c1a0902c41bcf26371fffc921123143d1dbaef24cc7e412ac4 not found: ID does not exist" Oct 31 01:02:11.547588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547556 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.547721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.547721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547689 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.547850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547729 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.547850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547832 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.547960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547878 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.547960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.547910 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.548083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548013 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.548083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548078 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.548195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548093 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:11.548195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548150 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\") pod \"8a2ed75c-5330-4a56-8ed6-f359274dee68\" (UID: \"8a2ed75c-5330-4a56-8ed6-f359274dee68\") " Oct 31 01:02:11.548359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548310 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.548359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548333 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.548589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548556 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.548905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548882 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.548988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548911 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ed75c-5330-4a56-8ed6-f359274dee68-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.548988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548925 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.548988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548938 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.548988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.548951 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.550904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.550761 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.550904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.550825 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "kube-api-access-9xh9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:11.550904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.550836 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:11.550904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.550872 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo" pod "8a2ed75c-5330-4a56-8ed6-f359274dee68" (UID: "8a2ed75c-5330-4a56-8ed6-f359274dee68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:11.650602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.650563 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xh9m\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-kube-api-access-9xh9m\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.650602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.650601 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.650791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.650614 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.650791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.650626 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ed75c-5330-4a56-8ed6-f359274dee68-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.843294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.843175 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 01:02:11.847182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:11.847150 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-2x457"] Oct 31 01:02:13.517746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.517711 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 01:02:13.518256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.517962 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerName="discovery" containerID="cri-o://a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593" gracePeriod=30 Oct 31 01:02:13.686023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.685991 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" path="/var/lib/kubelet/pods/8a2ed75c-5330-4a56-8ed6-f359274dee68/volumes" Oct 31 01:02:13.818126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.818101 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 01:02:13.882194 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882164 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882276 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882315 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882359 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882447 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882516 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.882684 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.882662 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\") pod \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\" (UID: \"3caa6982-ed49-43b3-b239-7d6c98ca7abd\") " Oct 31 01:02:13.885381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.885349 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:13.885760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.885737 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:13.887987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.887959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:13.888774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.888754 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:13.890258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.890187 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:13.906665 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.906628 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "kube-api-access-z6w6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:13.908172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.908144 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert" pod "3caa6982-ed49-43b3-b239-7d6c98ca7abd" (UID: "3caa6982-ed49-43b3-b239-7d6c98ca7abd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:13.984596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984565 5011 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-kubeconfig\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984594 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984603 5011 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-cacerts\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984611 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w6f\" (UniqueName: \"kubernetes.io/projected/3caa6982-ed49-43b3-b239-7d6c98ca7abd-kube-api-access-z6w6f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984622 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-dns-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984630 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3caa6982-ed49-43b3-b239-7d6c98ca7abd-istio-csr-ca-configmap\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:13.984803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:13.984642 5011 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3caa6982-ed49-43b3-b239-7d6c98ca7abd-local-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:14.522804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.522765 5011 generic.go:356] "Generic (PLEG): container finished" podID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerID="a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593" exitCode=0 Oct 31 01:02:14.523293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.522844 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" Oct 31 01:02:14.523293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.522849 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" event={"ID":"3caa6982-ed49-43b3-b239-7d6c98ca7abd","Type":"ContainerDied","Data":"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593"} Oct 31 01:02:14.523293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.522952 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69" event={"ID":"3caa6982-ed49-43b3-b239-7d6c98ca7abd","Type":"ContainerDied","Data":"9d51dec6ff3be5c80b327cf3da5f9679b13187f64b5112d92d8020f6dba48843"} Oct 31 01:02:14.523293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.522986 5011 scope.go:122] "RemoveContainer" containerID="a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593" Oct 31 01:02:14.539467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.539438 5011 scope.go:122] "RemoveContainer" containerID="a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593" Oct 31 01:02:14.539816 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:14.539787 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593\": container with ID starting with a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593 not found: ID does not exist" containerID="a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593" Oct 31 01:02:14.539947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.539829 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593"} err="failed to get container status \"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593\": rpc error: code = NotFound desc = could not find container \"a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593\": container with ID starting with a825c6f51223e3fac88ab816fe317b4051fcff19a7091098765b390c27028593 not found: ID does not exist" Oct 31 01:02:14.590856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.590828 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 01:02:14.602164 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:14.602141 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s7c69"] Oct 31 01:02:15.684282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:15.684246 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" path="/var/lib/kubelet/pods/3caa6982-ed49-43b3-b239-7d6c98ca7abd/volumes" Oct 31 01:02:19.761592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:19.761559 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:02:31.950919 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:31.950876 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:02:37.702717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:37.702679 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 01:02:37.703270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:37.702887 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" podUID="e55466fa-794d-4f46-87a0-78ed08b1ec89" containerName="maas-api" containerID="cri-o://70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15" gracePeriod=30 Oct 31 01:02:38.310229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.304568 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 01:02:38.379728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.379694 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\") pod \"e55466fa-794d-4f46-87a0-78ed08b1ec89\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " Oct 31 01:02:38.379980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.379961 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\") pod \"e55466fa-794d-4f46-87a0-78ed08b1ec89\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " Oct 31 01:02:38.380486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.380456 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\" (UniqueName: \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\") pod \"e55466fa-794d-4f46-87a0-78ed08b1ec89\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " Oct 31 01:02:38.380790 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.380768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\") pod \"e55466fa-794d-4f46-87a0-78ed08b1ec89\" (UID: \"e55466fa-794d-4f46-87a0-78ed08b1ec89\") " Oct 31 01:02:38.384589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.384564 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs" pod "e55466fa-794d-4f46-87a0-78ed08b1ec89" (UID: "e55466fa-794d-4f46-87a0-78ed08b1ec89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:38.385055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.385032 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca" pod "e55466fa-794d-4f46-87a0-78ed08b1ec89" (UID: "e55466fa-794d-4f46-87a0-78ed08b1ec89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:38.388606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.388569 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz" pod "e55466fa-794d-4f46-87a0-78ed08b1ec89" (UID: "e55466fa-794d-4f46-87a0-78ed08b1ec89"). InnerVolumeSpecName "kube-api-access-srwmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:38.414455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.414418 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls" pod "e55466fa-794d-4f46-87a0-78ed08b1ec89" (UID: "e55466fa-794d-4f46-87a0-78ed08b1ec89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:38.483072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.482983 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwmz\" (UniqueName: \"kubernetes.io/projected/e55466fa-794d-4f46-87a0-78ed08b1ec89-kube-api-access-srwmz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:38.483072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.483014 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:38.483072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.483029 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55466fa-794d-4f46-87a0-78ed08b1ec89-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:38.483072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.483042 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55466fa-794d-4f46-87a0-78ed08b1ec89-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:38.771262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.771111 5011 generic.go:356] "Generic (PLEG): container finished" podID="e55466fa-794d-4f46-87a0-78ed08b1ec89" containerID="70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15" exitCode=0 Oct 31 01:02:38.771262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.771169 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" event={"ID":"e55466fa-794d-4f46-87a0-78ed08b1ec89","Type":"ContainerDied","Data":"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15"} Oct 31 01:02:38.771262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.771241 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" Oct 31 01:02:38.786571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.786532 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw" event={"ID":"e55466fa-794d-4f46-87a0-78ed08b1ec89","Type":"ContainerDied","Data":"15ef8ca5b7bbd8a37146f52ae95d5fec3a01fc02a769e0f2d040ae866813a8d6"} Oct 31 01:02:38.786779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.786614 5011 scope.go:122] "RemoveContainer" containerID="70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15" Oct 31 01:02:38.799608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.799583 5011 scope.go:122] "RemoveContainer" containerID="70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15" Oct 31 01:02:38.799906 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:38.799879 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15\": container with ID starting with 70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15 not found: ID does not exist" containerID="70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15" Oct 31 01:02:38.799971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.799915 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15"} err="failed to get container status \"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15\": rpc error: code = NotFound desc = could not find container \"70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15\": container with ID starting with 70e6f7b0cc67909c56bac796d1f580e25dd6d42690c81e946d8827f3b0e6ce15 not found: ID does not exist" Oct 31 01:02:38.835174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.835133 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 01:02:38.844537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:38.844505 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-609843-599df96497-5drnw"] Oct 31 01:02:39.688654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:39.688614 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55466fa-794d-4f46-87a0-78ed08b1ec89" path="/var/lib/kubelet/pods/e55466fa-794d-4f46-87a0-78ed08b1ec89/volumes" Oct 31 01:02:45.958227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:45.955288 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 01:02:45.958227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:45.955918 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" containerID="cri-o://8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097" gracePeriod=30 Oct 31 01:02:47.802268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.802235 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg"] Oct 31 01:02:47.803015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.802991 5011 cpu_manager.go:414] "containerMap: removing container" podUID="da76ea30-ff43-4aed-ac48-5483bc01983c" containerName="maas-api" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803014 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da76ea30-ff43-4aed-ac48-5483bc01983c" containerName="maas-api" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803031 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803038 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803054 5011 cpu_manager.go:414] "containerMap: removing container" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803061 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803074 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803082 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803096 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.803131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803105 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803258 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803272 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803293 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803302 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803312 5011 cpu_manager.go:414] "containerMap: removing container" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803318 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803337 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803376 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803390 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803397 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803419 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803457 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803474 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerName="discovery" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803481 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerName="discovery" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803527 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803536 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803545 5011 cpu_manager.go:414] "containerMap: removing container" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803552 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803581 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e55466fa-794d-4f46-87a0-78ed08b1ec89" containerName="maas-api" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803589 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55466fa-794d-4f46-87a0-78ed08b1ec89" containerName="maas-api" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803604 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" containerName="main" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803611 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" containerName="main" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803634 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803646 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803665 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803672 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803686 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803694 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.803695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803705 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803712 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803724 5011 cpu_manager.go:414] "containerMap: removing container" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803731 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803751 5011 cpu_manager.go:414] "containerMap: removing container" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803759 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803771 5011 cpu_manager.go:414] "containerMap: removing container" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803777 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803790 5011 cpu_manager.go:414] "containerMap: removing container" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803798 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803810 5011 cpu_manager.go:414] "containerMap: removing container" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803817 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803827 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ad99619e-203e-41b8-a029-6b434053aae1" containerName="revoke-keys" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803834 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad99619e-203e-41b8-a029-6b434053aae1" containerName="revoke-keys" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803848 5011 cpu_manager.go:414] "containerMap: removing container" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803855 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803869 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.803875 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804064 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804077 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804087 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3caa6982-ed49-43b3-b239-7d6c98ca7abd" containerName="discovery" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804098 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804107 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804116 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55466fa-794d-4f46-87a0-78ed08b1ec89" containerName="maas-api" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804125 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804133 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a066ab6d-e8ed-44bc-addc-b0e5859462d5" containerName="main" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804145 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804155 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad55bd2-5eae-4763-b439-c254281da697" containerName="istio-proxy" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804165 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804173 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804182 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804192 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804199 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="53214e19-073f-4c12-928b-51799031a596" containerName="istio-proxy" Oct 31 01:02:47.805291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.804333 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805379 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad99619e-203e-41b8-a029-6b434053aae1" containerName="revoke-keys" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805395 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805407 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805416 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805427 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805435 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805442 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805453 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="da76ea30-ff43-4aed-ac48-5483bc01983c" containerName="maas-api" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805464 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b95a10-5934-4538-806a-a932441beca7" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805482 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2ed75c-5330-4a56-8ed6-f359274dee68" containerName="istio-proxy" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805494 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805506 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:47.806974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.805516 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:47.811817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.811785 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg"] Oct 31 01:02:47.811945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.811906 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:47.848289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.848195 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:02:47.848289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.848260 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:02:47.933959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.933925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:47.934158 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.934133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:47.934302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:47.934286 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.036008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.035968 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.037117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.037092 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.037553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.037480 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.038062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.036980 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.041412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.041390 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.076380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.076350 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\") pod \"maas-api-revoke-sub-25947889-zpjlg\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.134434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.134411 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:48.498363 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:02:48.498329 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14467936_4c55_4cf2_8ac6_8ba82dff4510.slice/crio-706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f WatchSource:0}: Error finding container 706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f: Status 404 returned error can't find the container with id 706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f Oct 31 01:02:48.499726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.499693 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg"] Oct 31 01:02:48.855702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:48.855659 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" event={"ID":"14467936-4c55-4cf2-8ac6-8ba82dff4510","Type":"ContainerStarted","Data":"706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f"} Oct 31 01:02:49.870938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:49.870903 5011 generic.go:356] "Generic (PLEG): container finished" podID="14467936-4c55-4cf2-8ac6-8ba82dff4510" containerID="4272c1bddb10f89a410d33026ad0ea79507e04e9fcdae5038b6cbe1fd31f8845" exitCode=0 Oct 31 01:02:49.871441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:49.870971 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" event={"ID":"14467936-4c55-4cf2-8ac6-8ba82dff4510","Type":"ContainerDied","Data":"4272c1bddb10f89a410d33026ad0ea79507e04e9fcdae5038b6cbe1fd31f8845"} Oct 31 01:02:51.070822 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.070796 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:51.185501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.185423 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\") pod \"14467936-4c55-4cf2-8ac6-8ba82dff4510\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " Oct 31 01:02:51.185501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.185477 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\") pod \"14467936-4c55-4cf2-8ac6-8ba82dff4510\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " Oct 31 01:02:51.185763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.185743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\") pod \"14467936-4c55-4cf2-8ac6-8ba82dff4510\" (UID: \"14467936-4c55-4cf2-8ac6-8ba82dff4510\") " Oct 31 01:02:51.185949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.185848 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca" pod "14467936-4c55-4cf2-8ac6-8ba82dff4510" (UID: "14467936-4c55-4cf2-8ac6-8ba82dff4510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:51.186502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.186479 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.187789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.187761 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca" pod "14467936-4c55-4cf2-8ac6-8ba82dff4510" (UID: "14467936-4c55-4cf2-8ac6-8ba82dff4510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:51.187955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.187930 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m" pod "14467936-4c55-4cf2-8ac6-8ba82dff4510" (UID: "14467936-4c55-4cf2-8ac6-8ba82dff4510"). InnerVolumeSpecName "kube-api-access-bq78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:51.220807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.220784 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 01:02:51.287198 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287168 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.287378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287266 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.287378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287297 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.287378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287354 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.287552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287383 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.287552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287467 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.287552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287495 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.287552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287512 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287639 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287673 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287737 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.287766 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\") pod \"45238fc7-9ab2-4e80-8da0-72484cc14728\" (UID: \"45238fc7-9ab2-4e80-8da0-72484cc14728\") " Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288080 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288480 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288503 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288515 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45238fc7-9ab2-4e80-8da0-72484cc14728-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288528 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/14467936-4c55-4cf2-8ac6-8ba82dff4510-kube-api-access-bq78m\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288540 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14467936-4c55-4cf2-8ac6-8ba82dff4510-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288552 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.288967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.288755 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.291236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.290991 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.291236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.291042 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:51.292121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.292082 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "kube-api-access-p755d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:51.292575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.292549 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo" pod "45238fc7-9ab2-4e80-8da0-72484cc14728" (UID: "45238fc7-9ab2-4e80-8da0-72484cc14728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:51.389942 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.389911 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.389942 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.389941 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.390112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.389955 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p755d\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-kube-api-access-p755d\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.390112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.389967 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.390112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.389975 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45238fc7-9ab2-4e80-8da0-72484cc14728-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:02:51.889271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.889198 5011 generic.go:356] "Generic (PLEG): container finished" podID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerID="8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097" exitCode=0 Oct 31 01:02:51.889271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.889258 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" Oct 31 01:02:51.889512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.889288 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerDied","Data":"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097"} Oct 31 01:02:51.889512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.889345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz" event={"ID":"45238fc7-9ab2-4e80-8da0-72484cc14728","Type":"ContainerDied","Data":"77d46e9c97bcfff633c53c3edd59ec94f97d724ea4bc2493b5ae387c14c0e4a7"} Oct 31 01:02:51.889512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.889367 5011 scope.go:122] "RemoveContainer" containerID="8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097" Oct 31 01:02:51.890958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.890938 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" event={"ID":"14467936-4c55-4cf2-8ac6-8ba82dff4510","Type":"ContainerDied","Data":"706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f"} Oct 31 01:02:51.891051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.890958 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg" Oct 31 01:02:51.891051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.890972 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706dbe6b7b2007def9765d27a29d62597200c65a59acc78a412d0c146dd7045f" Oct 31 01:02:51.903044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.903027 5011 scope.go:122] "RemoveContainer" containerID="46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1" Oct 31 01:02:51.914390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.914367 5011 scope.go:122] "RemoveContainer" containerID="8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097" Oct 31 01:02:51.914665 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:51.914647 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097\": container with ID starting with 8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097 not found: ID does not exist" containerID="8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097" Oct 31 01:02:51.914734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.914674 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097"} err="failed to get container status \"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097\": rpc error: code = NotFound desc = could not find container \"8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097\": container with ID starting with 8910f21a88c40e6ada6ad6146b3212dc852a2d6fa68f7aaea185df5111a9b097 not found: ID does not exist" Oct 31 01:02:51.914734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.914691 5011 scope.go:122] "RemoveContainer" containerID="46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1" Oct 31 01:02:51.914951 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:02:51.914932 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1\": container with ID starting with 46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1 not found: ID does not exist" containerID="46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1" Oct 31 01:02:51.914995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.914957 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1"} err="failed to get container status \"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1\": rpc error: code = NotFound desc = could not find container \"46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1\": container with ID starting with 46d288ad3d46a241ebb4edce4ce7ca5b0df8139aa8f0a215b5468b6eb466d8f1 not found: ID does not exist" Oct 31 01:02:51.958562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.958492 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 01:02:51.971274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:51.971242 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-609843-openshift-default-db7868f45-jmbpz"] Oct 31 01:02:53.685407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:53.685368 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" path="/var/lib/kubelet/pods/45238fc7-9ab2-4e80-8da0-72484cc14728/volumes" Oct 31 01:02:58.652750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:58.652709 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 01:02:59.370752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.370689 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7"] Oct 31 01:02:59.371755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371729 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.371830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371761 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.371830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371781 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.371830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371789 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.371830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371807 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:59.371830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371815 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06da2-626f-4eea-97b1-1b47431c255c" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371834 5011 cpu_manager.go:414] "containerMap: removing container" podUID="14467936-4c55-4cf2-8ac6-8ba82dff4510" containerName="revoke-keys" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371841 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14467936-4c55-4cf2-8ac6-8ba82dff4510" containerName="revoke-keys" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371867 5011 cpu_manager.go:414] "containerMap: removing container" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371874 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45849c90-dd35-4665-8f92-df1e3a20aed0" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371893 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371902 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371915 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371922 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371936 5011 cpu_manager.go:414] "containerMap: removing container" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371943 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:59.371994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371993 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.371999 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372012 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372020 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372180 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea47c08-65fa-4182-951e-1e2e3295ce15" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372196 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a641442d-72f6-4cc7-a9be-01585637a854" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372234 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a8bd4e-7c2b-4a1b-b71f-fb8075d95aa7" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372244 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372250 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="45238fc7-9ab2-4e80-8da0-72484cc14728" containerName="istio-proxy" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372257 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4be7a7-e9a7-441f-ae8a-bfbd86daec90" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372264 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="14467936-4c55-4cf2-8ac6-8ba82dff4510" containerName="revoke-keys" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372271 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97a90a6-88a9-48fa-9c79-624fd9bd2b28" containerName="cleanup" Oct 31 01:02:59.372371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.372277 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b53a2a-bd43-4c27-8815-5f47bad4a011" containerName="cleanup" Oct 31 01:02:59.378397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.378371 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.394776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.394718 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7"] Oct 31 01:02:59.403797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.400617 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:02:59.403797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.400863 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:02:59.546283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.546246 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.546468 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.546385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5kk\" (UniqueName: \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.546540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.546489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.647982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.647910 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5kk\" (UniqueName: \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.647982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.647960 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.648182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.648135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.648825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.648797 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.650530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.650505 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.671846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.671822 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5kk\" (UniqueName: \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\") pod \"maas-api-revoke-sub-f7e2e6e8-tsdq7\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:02:59.695090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:02:59.695061 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:03:00.020655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:00.020627 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7"] Oct 31 01:03:00.021146 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:03:00.021114 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eaa2dc0_5aa3_4b4c_83de_f1618d232941.slice/crio-d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4 WatchSource:0}: Error finding container d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4: Status 404 returned error can't find the container with id d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4 Oct 31 01:03:00.978705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:00.978670 5011 generic.go:356] "Generic (PLEG): container finished" podID="2eaa2dc0-5aa3-4b4c-83de-f1618d232941" containerID="629844486f20a8465c7cda80ab427ae006cfb1bc5de63d431b5e5a5b1bc381f4" exitCode=0 Oct 31 01:03:00.979079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:00.978760 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" event={"ID":"2eaa2dc0-5aa3-4b4c-83de-f1618d232941","Type":"ContainerDied","Data":"629844486f20a8465c7cda80ab427ae006cfb1bc5de63d431b5e5a5b1bc381f4"} Oct 31 01:03:00.979079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:00.978857 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" event={"ID":"2eaa2dc0-5aa3-4b4c-83de-f1618d232941","Type":"ContainerStarted","Data":"d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4"} Oct 31 01:03:01.951346 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:03:01.951311 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:03:02.168462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.168434 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:03:02.185486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.185457 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\") pod \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " Oct 31 01:03:02.185619 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.185551 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\") pod \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " Oct 31 01:03:02.185728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.185704 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca" pod "2eaa2dc0-5aa3-4b4c-83de-f1618d232941" (UID: "2eaa2dc0-5aa3-4b4c-83de-f1618d232941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:02.185812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.185712 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\" (UniqueName: \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\") pod \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\" (UID: \"2eaa2dc0-5aa3-4b4c-83de-f1618d232941\") " Oct 31 01:03:02.186456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.186433 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:02.187812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.187790 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca" pod "2eaa2dc0-5aa3-4b4c-83de-f1618d232941" (UID: "2eaa2dc0-5aa3-4b4c-83de-f1618d232941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:02.187916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.187888 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk" pod "2eaa2dc0-5aa3-4b4c-83de-f1618d232941" (UID: "2eaa2dc0-5aa3-4b4c-83de-f1618d232941"). InnerVolumeSpecName "kube-api-access-2m5kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:02.287879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.287794 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:02.287879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.287833 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m5kk\" (UniqueName: \"kubernetes.io/projected/2eaa2dc0-5aa3-4b4c-83de-f1618d232941-kube-api-access-2m5kk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:02.995390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.995312 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" Oct 31 01:03:02.995635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.995313 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7" event={"ID":"2eaa2dc0-5aa3-4b4c-83de-f1618d232941","Type":"ContainerDied","Data":"d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4"} Oct 31 01:03:02.995635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:02.995437 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5db6b6d3b5294f0596ec1727bf2bcd046cc447e1708f8f2ebcc86e35c9270d4" Oct 31 01:03:13.678699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:13.678648 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerName="main" containerID="cri-o://f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb" gracePeriod=45 Oct 31 01:03:14.128444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.128407 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 01:03:14.153499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.153456 5011 generic.go:356] "Generic (PLEG): container finished" podID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerID="f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb" exitCode=0 Oct 31 01:03:14.153773 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.153740 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" event={"ID":"3951fb9f-26d6-43d2-b1af-91b2e11837ff","Type":"ContainerDied","Data":"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb"} Oct 31 01:03:14.153869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.153805 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" event={"ID":"3951fb9f-26d6-43d2-b1af-91b2e11837ff","Type":"ContainerDied","Data":"ad54b3b19dcbc9c95e12445e7c5da8ef3f92a83b9dc76a97e384137d6fd4df47"} Oct 31 01:03:14.153869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.153829 5011 scope.go:122] "RemoveContainer" containerID="f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb" Oct 31 01:03:14.166263 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.166242 5011 scope.go:122] "RemoveContainer" containerID="f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb" Oct 31 01:03:14.166613 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:03:14.166578 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb\": container with ID starting with f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb not found: ID does not exist" containerID="f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb" Oct 31 01:03:14.166673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.166625 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb"} err="failed to get container status \"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb\": rpc error: code = NotFound desc = could not find container \"f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb\": container with ID starting with f0d94490cdc5b67c7843fc2942df79bef269bda59cef3c88e25186a9098c01cb not found: ID does not exist" Oct 31 01:03:14.272153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272071 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\" (UniqueName: \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272116 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272425 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272354 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272418 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:14.272495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272430 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272516 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272640 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\") pod \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\" (UID: \"3951fb9f-26d6-43d2-b1af-91b2e11837ff\") " Oct 31 01:03:14.272669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272651 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:14.272817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.272791 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:14.273349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.273320 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.273486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.273378 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.273486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.273392 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.274739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.274710 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "kube-api-access-nvtls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:14.274739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.274718 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:14.274926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.274749 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm" pod "3951fb9f-26d6-43d2-b1af-91b2e11837ff" (UID: "3951fb9f-26d6-43d2-b1af-91b2e11837ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:14.374795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.374763 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtls\" (UniqueName: \"kubernetes.io/projected/3951fb9f-26d6-43d2-b1af-91b2e11837ff-kube-api-access-nvtls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.374941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.374800 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3951fb9f-26d6-43d2-b1af-91b2e11837ff-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.374941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.374812 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3951fb9f-26d6-43d2-b1af-91b2e11837ff-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:14.936806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:14.936770 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:03:15.220904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:15.220822 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc" Oct 31 01:03:15.277174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:15.277140 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 01:03:15.289283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:15.289255 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-762753/test-model-762753-kserve-56df5c8f9f-xbkmc"] Oct 31 01:03:15.685941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:15.685909 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" path="/var/lib/kubelet/pods/3951fb9f-26d6-43d2-b1af-91b2e11837ff/volumes" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.123179 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124089 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2eaa2dc0-5aa3-4b4c-83de-f1618d232941" containerName="revoke-keys" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124106 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eaa2dc0-5aa3-4b4c-83de-f1618d232941" containerName="revoke-keys" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124175 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerName="main" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124183 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerName="main" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124358 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3951fb9f-26d6-43d2-b1af-91b2e11837ff" containerName="main" Oct 31 01:03:30.126228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.124375 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eaa2dc0-5aa3-4b4c-83de-f1618d232941" containerName="revoke-keys" Oct 31 01:03:30.132667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.132643 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.157286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.157255 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:03:30.188299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.188271 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-b8fd5a-openshift-default-dockercfg-mwm8c\"" Oct 31 01:03:30.197633 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197600 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8qh\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197715 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197754 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197801 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197892 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.197974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.197927 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.198143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.198014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.198143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.198123 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299036 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299095 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299178 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299399 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299448 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8qh\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299479 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299514 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.299603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.299560 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.301177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.300043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.301177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.300321 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.301177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.300514 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.301177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.300756 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.316227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.303983 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.320895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.320770 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.336227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.331501 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.400275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.399649 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.405255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.401781 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8qh\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\") pod \"e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.457621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.455874 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:30.797997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.797959 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:03:30.798185 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:03:30.798162 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b878d08_8de0_4f73_9b8e_f0bffd236715.slice/crio-7742a792a5b2a41993660fa746cfb513ddd4eaaa9022008ce8215cd1d0fff4a0 WatchSource:0}: Error finding container 7742a792a5b2a41993660fa746cfb513ddd4eaaa9022008ce8215cd1d0fff4a0: Status 404 returned error can't find the container with id 7742a792a5b2a41993660fa746cfb513ddd4eaaa9022008ce8215cd1d0fff4a0 Oct 31 01:03:30.801054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.801023 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:30.801152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.801104 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:30.801152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:30.801133 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:31.370950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:31.370912 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerStarted","Data":"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074"} Oct 31 01:03:31.371497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:31.370975 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerStarted","Data":"7742a792a5b2a41993660fa746cfb513ddd4eaaa9022008ce8215cd1d0fff4a0"} Oct 31 01:03:31.430858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:31.430800 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podStartSLOduration=1.4307811670000001 podStartE2EDuration="1.430781167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:31.42884899 +0000 UTC m=+2679.877936630" watchObservedRunningTime="2026-10-31 01:03:31.430781167 +0000 UTC m=+2679.879868807" Oct 31 01:03:31.456390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:31.456352 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:31.463074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:31.463042 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:31.954228 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:03:31.953919 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:03:32.031361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.031322 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 01:03:32.379203 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.379172 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:32.380341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.380309 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:03:32.412883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.412856 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq"] Oct 31 01:03:32.417541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.417519 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.434621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.434598 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:32.447389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.446536 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:03:32.458002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.457699 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq"] Oct 31 01:03:32.555102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.555067 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crrw\" (UniqueName: \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.555311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.555286 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.555383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.555335 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.657338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.657250 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crrw\" (UniqueName: \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.657536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.657418 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.657536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.657466 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.658282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.658254 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.660637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.660611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.681802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.681769 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crrw\" (UniqueName: \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\") pod \"maas-api-revoke-keys-e2e-inf-a-762753-c6lkq\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:32.735036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:32.735003 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:33.060510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:33.060476 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq"] Oct 31 01:03:33.060697 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:03:33.060477 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16556ef_a7cc_4b42_bbbb_feceacb277a7.slice/crio-27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2 WatchSource:0}: Error finding container 27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2: Status 404 returned error can't find the container with id 27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2 Oct 31 01:03:33.388082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:33.388049 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" event={"ID":"f16556ef-a7cc-4b42-bbbb-feceacb277a7","Type":"ContainerStarted","Data":"27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2"} Oct 31 01:03:34.396963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:34.396925 5011 generic.go:356] "Generic (PLEG): container finished" podID="f16556ef-a7cc-4b42-bbbb-feceacb277a7" containerID="6d506cae6895ce02e65bccc7bacb7abbbd9157d9d5265b2bdb3b9779acd49252" exitCode=0 Oct 31 01:03:34.397491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:34.396966 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" event={"ID":"f16556ef-a7cc-4b42-bbbb-feceacb277a7","Type":"ContainerDied","Data":"6d506cae6895ce02e65bccc7bacb7abbbd9157d9d5265b2bdb3b9779acd49252"} Oct 31 01:03:35.670796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.670734 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:35.806128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.806097 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\") pod \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " Oct 31 01:03:35.806343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.806183 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\" (UniqueName: \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\") pod \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " Oct 31 01:03:35.806343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.806297 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\") pod \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\" (UID: \"f16556ef-a7cc-4b42-bbbb-feceacb277a7\") " Oct 31 01:03:35.806508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.806458 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca" pod "f16556ef-a7cc-4b42-bbbb-feceacb277a7" (UID: "f16556ef-a7cc-4b42-bbbb-feceacb277a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:35.807299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.807266 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:35.808398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.808374 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca" pod "f16556ef-a7cc-4b42-bbbb-feceacb277a7" (UID: "f16556ef-a7cc-4b42-bbbb-feceacb277a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:35.808482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.808457 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw" pod "f16556ef-a7cc-4b42-bbbb-feceacb277a7" (UID: "f16556ef-a7cc-4b42-bbbb-feceacb277a7"). InnerVolumeSpecName "kube-api-access-2crrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:35.908343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.908309 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crrw\" (UniqueName: \"kubernetes.io/projected/f16556ef-a7cc-4b42-bbbb-feceacb277a7-kube-api-access-2crrw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:35.908343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:35.908339 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16556ef-a7cc-4b42-bbbb-feceacb277a7-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:36.418034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:36.418001 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" Oct 31 01:03:36.418034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:36.418021 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq" event={"ID":"f16556ef-a7cc-4b42-bbbb-feceacb277a7","Type":"ContainerDied","Data":"27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2"} Oct 31 01:03:36.418311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:36.418083 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c3ecd45529db6dde3e3e7d78f34f344a2fdb03666bf1be269d7de6799d4ca2" Oct 31 01:03:38.767407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.767373 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc"] Oct 31 01:03:38.768004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.767989 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f16556ef-a7cc-4b42-bbbb-feceacb277a7" containerName="revoke-keys" Oct 31 01:03:38.768058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.768005 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16556ef-a7cc-4b42-bbbb-feceacb277a7" containerName="revoke-keys" Oct 31 01:03:38.768127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.768118 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16556ef-a7cc-4b42-bbbb-feceacb277a7" containerName="revoke-keys" Oct 31 01:03:38.772516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.772495 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.790372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.790343 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc"] Oct 31 01:03:38.797692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.797662 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:38.797812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.797707 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:03:38.879374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.879343 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.879540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.879510 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkmq\" (UniqueName: \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.879686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.879660 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.981590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.981557 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.981770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.981725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.981882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.981863 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkmq\" (UniqueName: \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.982538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.982510 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:38.984611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:38.984589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:39.005768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:39.005741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkmq\" (UniqueName: \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\") pod \"maas-api-revoke-sub-83eb2066-q6jkc\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:39.087128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:39.087096 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:39.431603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:39.431563 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc"] Oct 31 01:03:39.432834 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:03:39.432795 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6248d3_1ec4_4a86_a46d_e3b7f0b2b399.slice/crio-11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2 WatchSource:0}: Error finding container 11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2: Status 404 returned error can't find the container with id 11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2 Oct 31 01:03:39.468972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:39.467984 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" event={"ID":"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399","Type":"ContainerStarted","Data":"11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2"} Oct 31 01:03:40.477119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:40.477086 5011 generic.go:356] "Generic (PLEG): container finished" podID="6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" containerID="c5ff85d1c4373bfbadb4deb350a6cef0bbb4b61f8ef3ee028f6cc7ac621a4367" exitCode=0 Oct 31 01:03:40.477501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:40.477176 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" event={"ID":"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399","Type":"ContainerDied","Data":"c5ff85d1c4373bfbadb4deb350a6cef0bbb4b61f8ef3ee028f6cc7ac621a4367"} Oct 31 01:03:41.706451 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.706423 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:41.823073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.823030 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\") pod \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " Oct 31 01:03:41.823279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.823122 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\" (UniqueName: \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\") pod \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " Oct 31 01:03:41.823375 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.823356 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\") pod \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\" (UID: \"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399\") " Oct 31 01:03:41.823439 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.823406 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca" pod "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" (UID: "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:41.824976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.824948 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:41.825695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.825665 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq" pod "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" (UID: "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399"). InnerVolumeSpecName "kube-api-access-2kkmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.825792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.825757 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca" pod "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" (UID: "6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:41.926540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.926505 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:41.926540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:41.926539 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkmq\" (UniqueName: \"kubernetes.io/projected/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399-kube-api-access-2kkmq\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:42.495301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:42.495267 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" event={"ID":"6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399","Type":"ContainerDied","Data":"11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2"} Oct 31 01:03:42.495501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:42.495307 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc" Oct 31 01:03:42.495501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:42.495318 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a21d74f5d4450a53f0e4f997078cef66927ed8f8738a057b9d5fb5ce5f8cf2" Oct 31 01:03:47.054283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.054233 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" podUID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" containerName="main" containerID="cri-o://6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388" gracePeriod=45 Oct 31 01:03:47.342408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.342381 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 01:03:47.521398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521365 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.521587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521418 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\" (UniqueName: \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.521587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.521587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.521759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521639 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.521759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521636 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:47.521759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.521715 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\") pod \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\" (UID: \"5fb42f14-e7fe-46ad-abd6-3019fd679c73\") " Oct 31 01:03:47.522080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.522055 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:47.522380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.522361 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.522472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.522389 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.522472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.522453 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:47.523879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.523854 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:47.524113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.524085 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:47.524271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.524127 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td" pod "5fb42f14-e7fe-46ad-abd6-3019fd679c73" (UID: "5fb42f14-e7fe-46ad-abd6-3019fd679c73"). InnerVolumeSpecName "kube-api-access-hp8td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:47.540771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.540743 5011 generic.go:356] "Generic (PLEG): container finished" podID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" containerID="6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388" exitCode=0 Oct 31 01:03:47.540886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.540790 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" Oct 31 01:03:47.540886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.540830 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" event={"ID":"5fb42f14-e7fe-46ad-abd6-3019fd679c73","Type":"ContainerDied","Data":"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388"} Oct 31 01:03:47.540886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.540878 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm" event={"ID":"5fb42f14-e7fe-46ad-abd6-3019fd679c73","Type":"ContainerDied","Data":"eaed738deb0b7a911fa08d9540f32cedf4af0a9ca4ffc0b51c0c8a6a28c433f9"} Oct 31 01:03:47.541063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.540900 5011 scope.go:122] "RemoveContainer" containerID="6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388" Oct 31 01:03:47.553668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.553646 5011 scope.go:122] "RemoveContainer" containerID="6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388" Oct 31 01:03:47.553970 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:03:47.553950 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388\": container with ID starting with 6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388 not found: ID does not exist" containerID="6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388" Oct 31 01:03:47.554039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.553981 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388"} err="failed to get container status \"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388\": rpc error: code = NotFound desc = could not find container \"6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388\": container with ID starting with 6ca5c64bfe01bedd6831ee713572d50a012bdcd0273432466c089262a0ec1388 not found: ID does not exist" Oct 31 01:03:47.613270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.613242 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 01:03:47.620656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.620636 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9de7f5/test-model-9de7f5-kserve-7f9559854c-rwtzm"] Oct 31 01:03:47.623567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.623546 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.623567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.623566 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.623748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.623593 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb42f14-e7fe-46ad-abd6-3019fd679c73-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.623748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.623603 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8td\" (UniqueName: \"kubernetes.io/projected/5fb42f14-e7fe-46ad-abd6-3019fd679c73-kube-api-access-hp8td\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:03:47.684389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:47.684360 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" path="/var/lib/kubelet/pods/5fb42f14-e7fe-46ad-abd6-3019fd679c73/volumes" Oct 31 01:03:52.370584 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:52.370553 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:03:52.400662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:52.400638 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:03:58.918265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:58.918224 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq"] Oct 31 01:03:58.929801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:58.929770 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-762753-c6lkq"] Oct 31 01:03:59.684607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:03:59.684573 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16556ef-a7cc-4b42-bbbb-feceacb277a7" path="/var/lib/kubelet/pods/f16556ef-a7cc-4b42-bbbb-feceacb277a7/volumes" Oct 31 01:04:01.949845 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:04:01.949806 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:04:05.481726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:05.481674 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 01:04:06.025229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.022170 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:04:06.207125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207089 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc"] Oct 31 01:04:06.207748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207730 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" containerName="main" Oct 31 01:04:06.207798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207749 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" containerName="main" Oct 31 01:04:06.207798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207780 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" containerName="revoke-keys" Oct 31 01:04:06.207798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207786 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" containerName="revoke-keys" Oct 31 01:04:06.207894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207890 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb42f14-e7fe-46ad-abd6-3019fd679c73" containerName="main" Oct 31 01:04:06.207932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.207902 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" containerName="revoke-keys" Oct 31 01:04:06.213571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.213544 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.231491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.231466 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:06.231882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.231863 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:04:06.233668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.233645 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc"] Oct 31 01:04:06.349696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.349659 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.349930 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.349909 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6csx\" (UniqueName: \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.350054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.349963 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.451570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.451531 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.451766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.451628 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.451903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.451880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6csx\" (UniqueName: \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.452386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.452359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.454105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.454076 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.478741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.478711 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6csx\" (UniqueName: \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\") pod \"maas-api-revoke-sub-c658ebe8-f5bnc\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.528199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.528170 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:06.895026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.894993 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:04:06.895232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:06.895189 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" containerID="cri-o://65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8" gracePeriod=30 Oct 31 01:04:07.147236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:07.130114 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc"] Oct 31 01:04:07.828808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:07.828770 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" event={"ID":"af89544b-7560-43d4-987b-ccf30874f3ec","Type":"ContainerStarted","Data":"f8097b42ef145aef7487907bc7d2f06cd55ca7ca4b665e4d0311f07039cd68dc"} Oct 31 01:04:07.829293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:07.828826 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" event={"ID":"af89544b-7560-43d4-987b-ccf30874f3ec","Type":"ContainerStarted","Data":"40ee656e26622175729fe6f64a6f14a65ed308a730f30356a39b9e12b59f0543"} Oct 31 01:04:07.876872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:07.876815 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" podStartSLOduration=1.876802686 podStartE2EDuration="1.876802686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:07.876319058 +0000 UTC m=+2716.325406722" watchObservedRunningTime="2026-10-31 01:04:07.876802686 +0000 UTC m=+2716.325890356" Oct 31 01:04:08.837998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:08.837956 5011 generic.go:356] "Generic (PLEG): container finished" podID="af89544b-7560-43d4-987b-ccf30874f3ec" containerID="f8097b42ef145aef7487907bc7d2f06cd55ca7ca4b665e4d0311f07039cd68dc" exitCode=0 Oct 31 01:04:08.838478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:08.838036 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" event={"ID":"af89544b-7560-43d4-987b-ccf30874f3ec","Type":"ContainerDied","Data":"f8097b42ef145aef7487907bc7d2f06cd55ca7ca4b665e4d0311f07039cd68dc"} Oct 31 01:04:10.011982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.011949 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:10.033476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.033440 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\" (UniqueName: \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\") pod \"af89544b-7560-43d4-987b-ccf30874f3ec\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " Oct 31 01:04:10.033644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.033567 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\") pod \"af89544b-7560-43d4-987b-ccf30874f3ec\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " Oct 31 01:04:10.033793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.033769 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\") pod \"af89544b-7560-43d4-987b-ccf30874f3ec\" (UID: \"af89544b-7560-43d4-987b-ccf30874f3ec\") " Oct 31 01:04:10.034656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.034626 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca" pod "af89544b-7560-43d4-987b-ccf30874f3ec" (UID: "af89544b-7560-43d4-987b-ccf30874f3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:10.036266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.036237 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx" pod "af89544b-7560-43d4-987b-ccf30874f3ec" (UID: "af89544b-7560-43d4-987b-ccf30874f3ec"). InnerVolumeSpecName "kube-api-access-t6csx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:10.036752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.036727 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca" pod "af89544b-7560-43d4-987b-ccf30874f3ec" (UID: "af89544b-7560-43d4-987b-ccf30874f3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:10.136093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.135990 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:10.136093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.136026 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af89544b-7560-43d4-987b-ccf30874f3ec-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:10.136093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.136041 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6csx\" (UniqueName: \"kubernetes.io/projected/af89544b-7560-43d4-987b-ccf30874f3ec-kube-api-access-t6csx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:10.855651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.855620 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" Oct 31 01:04:10.855824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.855653 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc" event={"ID":"af89544b-7560-43d4-987b-ccf30874f3ec","Type":"ContainerDied","Data":"40ee656e26622175729fe6f64a6f14a65ed308a730f30356a39b9e12b59f0543"} Oct 31 01:04:10.855824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:10.855698 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ee656e26622175729fe6f64a6f14a65ed308a730f30356a39b9e12b59f0543" Oct 31 01:04:12.203079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.203053 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:04:12.266549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266512 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.266756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266657 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.266756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266704 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.266756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.266923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266790 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.266923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266819 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.266923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266891 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.267076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266925 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.267076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266975 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.267076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.266994 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.267076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267012 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\") pod \"d1a83023-2dad-4d0d-a728-77e377e1afb9\" (UID: \"d1a83023-2dad-4d0d-a728-77e377e1afb9\") " Oct 31 01:04:12.267258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267238 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:12.267666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267640 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.267858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267843 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.267987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267974 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a83023-2dad-4d0d-a728-77e377e1afb9-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.268108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267661 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.268108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.267731 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.269317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.269293 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.271460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.271433 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:12.271621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.271533 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:04:12.271918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.271894 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt" pod "d1a83023-2dad-4d0d-a728-77e377e1afb9" (UID: "d1a83023-2dad-4d0d-a728-77e377e1afb9"). InnerVolumeSpecName "kube-api-access-d5nkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:12.369253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369188 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5nkt\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-kube-api-access-d5nkt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.369253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369255 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.369429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369266 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.369429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369275 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.369429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369284 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.369429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.369293 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a83023-2dad-4d0d-a728-77e377e1afb9-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.872199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.872166 5011 generic.go:356] "Generic (PLEG): container finished" podID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerID="65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8" exitCode=0 Oct 31 01:04:12.872199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.872227 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" Oct 31 01:04:12.872508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.872235 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" event={"ID":"d1a83023-2dad-4d0d-a728-77e377e1afb9","Type":"ContainerDied","Data":"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8"} Oct 31 01:04:12.872508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.872292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg" event={"ID":"d1a83023-2dad-4d0d-a728-77e377e1afb9","Type":"ContainerDied","Data":"7783209aac66d301ab96012b0aa3d0f3aaf611fa68f481d4fae6967f3bfb214a"} Oct 31 01:04:12.872508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.872315 5011 scope.go:122] "RemoveContainer" containerID="65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8" Oct 31 01:04:12.884879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.884862 5011 scope.go:122] "RemoveContainer" containerID="65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8" Oct 31 01:04:12.885173 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:04:12.885152 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8\": container with ID starting with 65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8 not found: ID does not exist" containerID="65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8" Oct 31 01:04:12.885290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.885186 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8"} err="failed to get container status \"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8\": rpc error: code = NotFound desc = could not find container \"65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8\": container with ID starting with 65c33fc95f0204c627d2f47e5fadf7d5aa9a0df200f513ebd349c6591e3885e8 not found: ID does not exist" Oct 31 01:04:12.928050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.928016 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:04:12.939910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:12.939880 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-6fxtg"] Oct 31 01:04:13.684503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:13.684472 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" path="/var/lib/kubelet/pods/d1a83023-2dad-4d0d-a728-77e377e1afb9/volumes" Oct 31 01:04:17.883157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883120 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:04:17.883785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883768 5011 cpu_manager.go:414] "containerMap: removing container" podUID="af89544b-7560-43d4-987b-ccf30874f3ec" containerName="revoke-keys" Oct 31 01:04:17.883835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883786 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af89544b-7560-43d4-987b-ccf30874f3ec" containerName="revoke-keys" Oct 31 01:04:17.883835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883828 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" Oct 31 01:04:17.883905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883836 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" Oct 31 01:04:17.883951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883942 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a83023-2dad-4d0d-a728-77e377e1afb9" containerName="istio-proxy" Oct 31 01:04:17.883987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.883957 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="af89544b-7560-43d4-987b-ccf30874f3ec" containerName="revoke-keys" Oct 31 01:04:17.889073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.889055 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.909933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.909902 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:04:17.916833 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.916806 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-99a5c2-openshift-default-dockercfg-6l7wr\"" Oct 31 01:04:17.942828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.942798 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.942978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.942948 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943106 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943192 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943165 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ckz\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943289 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943324 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943398 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943453 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:17.943502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:17.943483 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.044988 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045032 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045258 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045373 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045423 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ckz\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045465 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045492 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045531 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045576 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045579 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045826 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.045946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.045924 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.046419 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.046395 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.047696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.047670 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.047828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.047811 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.069585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.069551 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.070849 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.070827 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ckz\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.203958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.203869 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:18.567647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.567604 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:04:18.568909 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:18.568883 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3019ac_6db7_4407_9e3b_4329b9de1df5.slice/crio-70593d22b10970d132c2e3149923afdbf6bbb070220f8ab2c4f9c4de5f5b409a WatchSource:0}: Error finding container 70593d22b10970d132c2e3149923afdbf6bbb070220f8ab2c4f9c4de5f5b409a: Status 404 returned error can't find the container with id 70593d22b10970d132c2e3149923afdbf6bbb070220f8ab2c4f9c4de5f5b409a Oct 31 01:04:18.572582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.572545 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:18.572678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.572617 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:18.572678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.572647 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:18.928990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.928062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" event={"ID":"cb3019ac-6db7-4407-9e3b-4329b9de1df5","Type":"ContainerStarted","Data":"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec"} Oct 31 01:04:18.928990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.928116 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" event={"ID":"cb3019ac-6db7-4407-9e3b-4329b9de1df5","Type":"ContainerStarted","Data":"70593d22b10970d132c2e3149923afdbf6bbb070220f8ab2c4f9c4de5f5b409a"} Oct 31 01:04:18.987743 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:18.987679 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podStartSLOduration=1.987659681 podStartE2EDuration="1.987659681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:18.98405431 +0000 UTC m=+2727.433141950" watchObservedRunningTime="2026-10-31 01:04:18.987659681 +0000 UTC m=+2727.436747324" Oct 31 01:04:19.204649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:19.204567 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:19.206332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:19.206300 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:19.206489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:19.206368 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:19.995071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:19.995024 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 01:04:20.175298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.171134 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk"] Oct 31 01:04:20.205232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.194422 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.205232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.204614 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:20.205232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.204667 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:20.216240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.212471 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:20.216240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.213176 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:04:20.238234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.237022 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk"] Oct 31 01:04:20.270363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.255380 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztm8k\" (UniqueName: \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.270363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.255446 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.270363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.255502 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.361241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.359339 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztm8k\" (UniqueName: \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.361241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.359408 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.361241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.359464 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.364239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.361632 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.380861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.380832 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.387863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.387822 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztm8k\" (UniqueName: \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\") pod \"maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.513198 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.513153 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" podUID="b2316e45-72e5-4a67-af6a-4c765bf1501d" containerName="main" containerID="cri-o://f8c4a27aa48dfbda51b66d6873325ede229e71e60f9a70fb246486d8ff8653e4" gracePeriod=45 Oct 31 01:04:20.589640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.589612 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:20.987229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.986993 5011 generic.go:356] "Generic (PLEG): container finished" podID="b2316e45-72e5-4a67-af6a-4c765bf1501d" containerID="f8c4a27aa48dfbda51b66d6873325ede229e71e60f9a70fb246486d8ff8653e4" exitCode=0 Oct 31 01:04:20.993250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:20.987931 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" event={"ID":"b2316e45-72e5-4a67-af6a-4c765bf1501d","Type":"ContainerDied","Data":"f8c4a27aa48dfbda51b66d6873325ede229e71e60f9a70fb246486d8ff8653e4"} Oct 31 01:04:21.052268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.051366 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk"] Oct 31 01:04:21.053380 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:21.053260 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a9ad90_f81f_43bc_bf06_2bb0620592e7.slice/crio-df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff WatchSource:0}: Error finding container df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff: Status 404 returned error can't find the container with id df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff Oct 31 01:04:21.055828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.055807 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 01:04:21.082496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082471 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.082609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082522 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.082674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082617 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\" (UniqueName: \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.082734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082704 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.082869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082849 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082851 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.082987 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\") pod \"b2316e45-72e5-4a67-af6a-4c765bf1501d\" (UID: \"b2316e45-72e5-4a67-af6a-4c765bf1501d\") " Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.083022 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.083041 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.083553 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.083569 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.083851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.083578 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.084804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.084776 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:21.085160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.085141 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "kube-api-access-w94md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:21.085980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.085961 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm" pod "b2316e45-72e5-4a67-af6a-4c765bf1501d" (UID: "b2316e45-72e5-4a67-af6a-4c765bf1501d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:21.184767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.184688 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2316e45-72e5-4a67-af6a-4c765bf1501d-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.184767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.184716 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2316e45-72e5-4a67-af6a-4c765bf1501d-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.184767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.184726 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94md\" (UniqueName: \"kubernetes.io/projected/b2316e45-72e5-4a67-af6a-4c765bf1501d-kube-api-access-w94md\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:21.205700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.205664 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:21.205864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.205726 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:21.998133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.998101 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" Oct 31 01:04:21.998369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.998108 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c" event={"ID":"b2316e45-72e5-4a67-af6a-4c765bf1501d","Type":"ContainerDied","Data":"3da56b1d718eaf6c11067b72bdba625817395e4f6cf7cfa452ccacf35e84ca7f"} Oct 31 01:04:21.998369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:21.998282 5011 scope.go:122] "RemoveContainer" containerID="f8c4a27aa48dfbda51b66d6873325ede229e71e60f9a70fb246486d8ff8653e4" Oct 31 01:04:22.000064 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.000035 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" event={"ID":"b9a9ad90-f81f-43bc-bf06-2bb0620592e7","Type":"ContainerStarted","Data":"44e3c903891b51d7d26bb99ff6707b92c1deeba758e381adb6b8c095541e2203"} Oct 31 01:04:22.000176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.000101 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" event={"ID":"b9a9ad90-f81f-43bc-bf06-2bb0620592e7","Type":"ContainerStarted","Data":"df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff"} Oct 31 01:04:22.063517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.063461 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" podStartSLOduration=2.063448138 podStartE2EDuration="2.063448138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:22.060288793 +0000 UTC m=+2730.509376433" watchObservedRunningTime="2026-10-31 01:04:22.063448138 +0000 UTC m=+2730.512535781" Oct 31 01:04:22.109048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.109009 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 01:04:22.113608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.113578 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b87c62/test-model-b87c62-kserve-659b46d979-x876c"] Oct 31 01:04:22.204255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.204188 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:22.204427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:22.204275 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:23.011059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:23.011020 5011 generic.go:356] "Generic (PLEG): container finished" podID="b9a9ad90-f81f-43bc-bf06-2bb0620592e7" containerID="44e3c903891b51d7d26bb99ff6707b92c1deeba758e381adb6b8c095541e2203" exitCode=0 Oct 31 01:04:23.011297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:23.011110 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" event={"ID":"b9a9ad90-f81f-43bc-bf06-2bb0620592e7","Type":"ContainerDied","Data":"44e3c903891b51d7d26bb99ff6707b92c1deeba758e381adb6b8c095541e2203"} Oct 31 01:04:23.204681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:23.204643 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:23.205139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:23.204698 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:23.686967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:23.686925 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2316e45-72e5-4a67-af6a-4c765bf1501d" path="/var/lib/kubelet/pods/b2316e45-72e5-4a67-af6a-4c765bf1501d/volumes" Oct 31 01:04:24.204609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.204545 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 01:04:24.204763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.204605 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 01:04:24.307034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.307007 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:24.520229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.495123 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:04:24.550167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.550079 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b2316e45-72e5-4a67-af6a-4c765bf1501d" containerName="main" Oct 31 01:04:24.591255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.575244 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2316e45-72e5-4a67-af6a-4c765bf1501d" containerName="main" Oct 31 01:04:24.591255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.575311 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b9a9ad90-f81f-43bc-bf06-2bb0620592e7" containerName="revoke-keys" Oct 31 01:04:24.591255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.575321 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a9ad90-f81f-43bc-bf06-2bb0620592e7" containerName="revoke-keys" Oct 31 01:04:24.591255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.575643 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2316e45-72e5-4a67-af6a-4c765bf1501d" containerName="main" Oct 31 01:04:24.591255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.575669 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a9ad90-f81f-43bc-bf06-2bb0620592e7" containerName="revoke-keys" Oct 31 01:04:24.615518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.615478 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\" (UniqueName: \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\") pod \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " Oct 31 01:04:24.615750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.615731 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\") pod \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " Oct 31 01:04:24.615853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.615794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\") pod \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\" (UID: \"b9a9ad90-f81f-43bc-bf06-2bb0620592e7\") " Oct 31 01:04:24.669234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.661457 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca" pod "b9a9ad90-f81f-43bc-bf06-2bb0620592e7" (UID: "b9a9ad90-f81f-43bc-bf06-2bb0620592e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:24.682234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.678842 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.709242 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.698486 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:04:24.755427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.720793 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca" pod "b9a9ad90-f81f-43bc-bf06-2bb0620592e7" (UID: "b9a9ad90-f81f-43bc-bf06-2bb0620592e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:24.755669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.725364 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k" pod "b9a9ad90-f81f-43bc-bf06-2bb0620592e7" (UID: "b9a9ad90-f81f-43bc-bf06-2bb0620592e7"). InnerVolumeSpecName "kube-api-access-ztm8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:24.755773 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.743261 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-328d61-openshift-default-dockercfg-5b75g\"" Oct 31 01:04:24.762274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.741621 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:24.865032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.864396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.864906 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.864965 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865119 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwcn\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865178 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865350 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865452 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865502 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865547 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.865708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865665 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:24.865708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.865682 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztm8k\" (UniqueName: \"kubernetes.io/projected/b9a9ad90-f81f-43bc-bf06-2bb0620592e7-kube-api-access-ztm8k\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:24.967030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.966986 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwcn\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967052 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967160 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967272 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967322 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967375 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967496 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967868 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967555 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.967868 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.967843 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.968481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.968453 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.968593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.968530 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.968916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.968891 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.970315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.970292 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.970415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.970391 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.997691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.997649 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:24.999413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:24.999378 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwcn\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:25.035231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.032983 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" event={"ID":"b9a9ad90-f81f-43bc-bf06-2bb0620592e7","Type":"ContainerDied","Data":"df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff"} Oct 31 01:04:25.035231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.033050 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df4874a324152ae5d9f0cf71f6d65b3cb7d889c576a7ff9d24c4bc1275daccff" Oct 31 01:04:25.035231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.033117 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk" Oct 31 01:04:25.116228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.111200 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:25.400849 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.400754 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:25.402072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.402036 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:25.403349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.403323 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:04:25.513111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.513080 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:04:25.514430 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:25.514398 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c611828_648f_4f9a_968c_1da2c3b03d84.slice/crio-326c0e6bc8d5406c0146e2bebd13522ce995da9bed0413349fe140acf8b2f54a WatchSource:0}: Error finding container 326c0e6bc8d5406c0146e2bebd13522ce995da9bed0413349fe140acf8b2f54a: Status 404 returned error can't find the container with id 326c0e6bc8d5406c0146e2bebd13522ce995da9bed0413349fe140acf8b2f54a Oct 31 01:04:25.519546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.518540 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:25.519546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.518641 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:25.519546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:25.518689 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:26.048325 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:26.048183 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerStarted","Data":"ab09e36446414bedbe1f0b4568de590f69c1ae8f032a69e09ec16ac8d3f6719c"} Oct 31 01:04:26.048325 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:26.048282 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerStarted","Data":"326c0e6bc8d5406c0146e2bebd13522ce995da9bed0413349fe140acf8b2f54a"} Oct 31 01:04:26.106240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:26.104225 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podStartSLOduration=2.104176009 podStartE2EDuration="2.104176009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:26.102493817 +0000 UTC m=+2734.551581457" watchObservedRunningTime="2026-10-31 01:04:26.104176009 +0000 UTC m=+2734.553263650" Oct 31 01:04:26.111731 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:26.111702 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:26.122859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:26.122834 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:27.065581 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:27.065552 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:27.068602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:27.068579 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:04:31.949501 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:04:31.949463 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:04:33.807771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.807697 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52"] Oct 31 01:04:33.814146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.814120 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:33.832398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.832367 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52"] Oct 31 01:04:33.838172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.838142 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:33.838405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.838156 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:04:33.963331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.963293 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xds7\" (UniqueName: \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:33.963518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.963444 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:33.963809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:33.963543 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.065357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.065274 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.065520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.065470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xds7\" (UniqueName: \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.065586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.065576 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.066303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.066263 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.067775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.067753 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.089503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.089478 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xds7\" (UniqueName: \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\") pod \"maas-api-revoke-sub-1f180056-s6t52\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.167550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.167518 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:34.477995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:34.477969 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52"] Oct 31 01:04:34.480154 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:34.480118 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b613dec_b1dc_4273_879c_1401953c6e6e.slice/crio-f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482 WatchSource:0}: Error finding container f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482: Status 404 returned error can't find the container with id f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482 Oct 31 01:04:35.168907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.168863 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" event={"ID":"8b613dec-b1dc-4273-879c-1401953c6e6e","Type":"ContainerStarted","Data":"bb82d90494a8ee076a09e47befc2e237e759b95e282e43f3ed536d0404def4c6"} Oct 31 01:04:35.169345 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.168931 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" event={"ID":"8b613dec-b1dc-4273-879c-1401953c6e6e","Type":"ContainerStarted","Data":"f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482"} Oct 31 01:04:35.212012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.211963 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" podStartSLOduration=2.211946748 podStartE2EDuration="2.211946748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:35.21038749 +0000 UTC m=+2743.659475129" watchObservedRunningTime="2026-10-31 01:04:35.211946748 +0000 UTC m=+2743.661034387" Oct 31 01:04:35.284182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.284146 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" podUID="157bb073-c882-4c53-9475-a8767ae5767d" containerName="main" containerID="cri-o://c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e" gracePeriod=45 Oct 31 01:04:35.579290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.579267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 01:04:35.687427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687392 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\" (UniqueName: \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687457 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687537 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687688 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687613 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687754 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687725 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\" (UniqueName: \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687743 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:35.687817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687750 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:35.687933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687858 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\") pod \"157bb073-c882-4c53-9475-a8767ae5767d\" (UID: \"157bb073-c882-4c53-9475-a8767ae5767d\") " Oct 31 01:04:35.687933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.687882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:35.688396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.688372 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:35.688396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.688398 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:35.688523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.688408 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:35.689785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.689763 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "kube-api-access-jclvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:35.689876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.689793 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:35.689933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.689914 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs" pod "157bb073-c882-4c53-9475-a8767ae5767d" (UID: "157bb073-c882-4c53-9475-a8767ae5767d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:35.789737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.789671 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclvn\" (UniqueName: \"kubernetes.io/projected/157bb073-c882-4c53-9475-a8767ae5767d-kube-api-access-jclvn\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:35.789737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.789700 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157bb073-c882-4c53-9475-a8767ae5767d-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:35.789737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:35.789711 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/157bb073-c882-4c53-9475-a8767ae5767d-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:36.178473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.178438 5011 generic.go:356] "Generic (PLEG): container finished" podID="8b613dec-b1dc-4273-879c-1401953c6e6e" containerID="bb82d90494a8ee076a09e47befc2e237e759b95e282e43f3ed536d0404def4c6" exitCode=0 Oct 31 01:04:36.178918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.178527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" event={"ID":"8b613dec-b1dc-4273-879c-1401953c6e6e","Type":"ContainerDied","Data":"bb82d90494a8ee076a09e47befc2e237e759b95e282e43f3ed536d0404def4c6"} Oct 31 01:04:36.179937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.179913 5011 generic.go:356] "Generic (PLEG): container finished" podID="157bb073-c882-4c53-9475-a8767ae5767d" containerID="c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e" exitCode=0 Oct 31 01:04:36.180098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.179939 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" event={"ID":"157bb073-c882-4c53-9475-a8767ae5767d","Type":"ContainerDied","Data":"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e"} Oct 31 01:04:36.180098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.179963 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" Oct 31 01:04:36.180098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.179980 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c" event={"ID":"157bb073-c882-4c53-9475-a8767ae5767d","Type":"ContainerDied","Data":"a281e4e3e462fda953c72b610f101db9d8e168cdbebf5d7c60ec94ecbba1296a"} Oct 31 01:04:36.180098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.179999 5011 scope.go:122] "RemoveContainer" containerID="c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e" Oct 31 01:04:36.192100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.192082 5011 scope.go:122] "RemoveContainer" containerID="c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e" Oct 31 01:04:36.192395 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:04:36.192374 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e\": container with ID starting with c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e not found: ID does not exist" containerID="c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e" Oct 31 01:04:36.192481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.192406 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e"} err="failed to get container status \"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e\": rpc error: code = NotFound desc = could not find container \"c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e\": container with ID starting with c3619dbefc339960dd06cb8ecd4984ab66aac6c78b9672ffd572dee5e8de933e not found: ID does not exist" Oct 31 01:04:36.267170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.267139 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 01:04:36.271059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:36.271033 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-01a117/test-model-01a117-kserve-84d96fcd9-d8w4c"] Oct 31 01:04:37.349542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.349508 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:37.511508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.511434 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\") pod \"8b613dec-b1dc-4273-879c-1401953c6e6e\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " Oct 31 01:04:37.511657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.511628 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\" (UniqueName: \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\") pod \"8b613dec-b1dc-4273-879c-1401953c6e6e\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " Oct 31 01:04:37.511701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.511686 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\") pod \"8b613dec-b1dc-4273-879c-1401953c6e6e\" (UID: \"8b613dec-b1dc-4273-879c-1401953c6e6e\") " Oct 31 01:04:37.512032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.512004 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca" pod "8b613dec-b1dc-4273-879c-1401953c6e6e" (UID: "8b613dec-b1dc-4273-879c-1401953c6e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:37.512314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.512299 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:37.513562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.513540 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca" pod "8b613dec-b1dc-4273-879c-1401953c6e6e" (UID: "8b613dec-b1dc-4273-879c-1401953c6e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:37.513704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.513684 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7" pod "8b613dec-b1dc-4273-879c-1401953c6e6e" (UID: "8b613dec-b1dc-4273-879c-1401953c6e6e"). InnerVolumeSpecName "kube-api-access-5xds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:37.613644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.613610 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b613dec-b1dc-4273-879c-1401953c6e6e-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:37.613644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.613641 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xds7\" (UniqueName: \"kubernetes.io/projected/8b613dec-b1dc-4273-879c-1401953c6e6e-kube-api-access-5xds7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:37.684105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:37.684072 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157bb073-c882-4c53-9475-a8767ae5767d" path="/var/lib/kubelet/pods/157bb073-c882-4c53-9475-a8767ae5767d/volumes" Oct 31 01:04:38.197482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:38.197456 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" Oct 31 01:04:38.197651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:38.197488 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52" event={"ID":"8b613dec-b1dc-4273-879c-1401953c6e6e","Type":"ContainerDied","Data":"f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482"} Oct 31 01:04:38.197651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:38.197540 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f489a67364a747ea6f12b5ee6062327b9d880b55f47acd5b0214a37bde8d7482" Oct 31 01:04:44.019707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.019671 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 01:04:44.020280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.019973 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" podUID="72be45f8-51e7-42e5-8097-25d7b6f56a33" containerName="maas-api" containerID="cri-o://87792358fdadb08116b4e7bea6f8681384186478a8000d1923d46b7145ca2e92" gracePeriod=30 Oct 31 01:04:44.393235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.392500 5011 generic.go:356] "Generic (PLEG): container finished" podID="72be45f8-51e7-42e5-8097-25d7b6f56a33" containerID="87792358fdadb08116b4e7bea6f8681384186478a8000d1923d46b7145ca2e92" exitCode=0 Oct 31 01:04:44.393235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.392720 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" event={"ID":"72be45f8-51e7-42e5-8097-25d7b6f56a33","Type":"ContainerDied","Data":"87792358fdadb08116b4e7bea6f8681384186478a8000d1923d46b7145ca2e92"} Oct 31 01:04:44.654521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.654443 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 01:04:44.727373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.724915 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\" (UniqueName: \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\") pod \"72be45f8-51e7-42e5-8097-25d7b6f56a33\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " Oct 31 01:04:44.727373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.725120 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\") pod \"72be45f8-51e7-42e5-8097-25d7b6f56a33\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " Oct 31 01:04:44.727373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.725555 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\") pod \"72be45f8-51e7-42e5-8097-25d7b6f56a33\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " Oct 31 01:04:44.727373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.725602 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\") pod \"72be45f8-51e7-42e5-8097-25d7b6f56a33\" (UID: \"72be45f8-51e7-42e5-8097-25d7b6f56a33\") " Oct 31 01:04:44.727373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.726752 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca" pod "72be45f8-51e7-42e5-8097-25d7b6f56a33" (UID: "72be45f8-51e7-42e5-8097-25d7b6f56a33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:44.735403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.735373 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs" pod "72be45f8-51e7-42e5-8097-25d7b6f56a33" (UID: "72be45f8-51e7-42e5-8097-25d7b6f56a33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:44.738511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.738340 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls" pod "72be45f8-51e7-42e5-8097-25d7b6f56a33" (UID: "72be45f8-51e7-42e5-8097-25d7b6f56a33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:44.738511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.738474 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n" pod "72be45f8-51e7-42e5-8097-25d7b6f56a33" (UID: "72be45f8-51e7-42e5-8097-25d7b6f56a33"). InnerVolumeSpecName "kube-api-access-9pk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:44.791127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.791084 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk"] Oct 31 01:04:44.795104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.795082 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b87c62-zsnfk"] Oct 31 01:04:44.828738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.828709 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:44.828887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.828750 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72be45f8-51e7-42e5-8097-25d7b6f56a33-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:44.828887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.828765 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be45f8-51e7-42e5-8097-25d7b6f56a33-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:44.828887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:44.828780 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pk4n\" (UniqueName: \"kubernetes.io/projected/72be45f8-51e7-42e5-8097-25d7b6f56a33-kube-api-access-9pk4n\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:45.402075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.402037 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" Oct 31 01:04:45.402545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.402074 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl" event={"ID":"72be45f8-51e7-42e5-8097-25d7b6f56a33","Type":"ContainerDied","Data":"8c7677122d21ae506ba54b335361c918382f5700b47bc6be442bb8f738078182"} Oct 31 01:04:45.402545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.402153 5011 scope.go:122] "RemoveContainer" containerID="87792358fdadb08116b4e7bea6f8681384186478a8000d1923d46b7145ca2e92" Oct 31 01:04:45.458961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.458933 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 01:04:45.470685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.470658 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b87c62-5f5df79bbd-wxzwl"] Oct 31 01:04:45.684461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.684389 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72be45f8-51e7-42e5-8097-25d7b6f56a33" path="/var/lib/kubelet/pods/72be45f8-51e7-42e5-8097-25d7b6f56a33/volumes" Oct 31 01:04:45.684841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:45.684827 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a9ad90-f81f-43bc-bf06-2bb0620592e7" path="/var/lib/kubelet/pods/b9a9ad90-f81f-43bc-bf06-2bb0620592e7/volumes" Oct 31 01:04:52.833381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.833336 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp"] Oct 31 01:04:52.833983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.833967 5011 cpu_manager.go:414] "containerMap: removing container" podUID="157bb073-c882-4c53-9475-a8767ae5767d" containerName="main" Oct 31 01:04:52.834033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.833984 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157bb073-c882-4c53-9475-a8767ae5767d" containerName="main" Oct 31 01:04:52.834033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834009 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8b613dec-b1dc-4273-879c-1401953c6e6e" containerName="revoke-keys" Oct 31 01:04:52.834033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834015 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b613dec-b1dc-4273-879c-1401953c6e6e" containerName="revoke-keys" Oct 31 01:04:52.834033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834028 5011 cpu_manager.go:414] "containerMap: removing container" podUID="72be45f8-51e7-42e5-8097-25d7b6f56a33" containerName="maas-api" Oct 31 01:04:52.834033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834034 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72be45f8-51e7-42e5-8097-25d7b6f56a33" containerName="maas-api" Oct 31 01:04:52.834191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834135 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="157bb073-c882-4c53-9475-a8767ae5767d" containerName="main" Oct 31 01:04:52.834191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834148 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b613dec-b1dc-4273-879c-1401953c6e6e" containerName="revoke-keys" Oct 31 01:04:52.834191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.834155 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="72be45f8-51e7-42e5-8097-25d7b6f56a33" containerName="maas-api" Oct 31 01:04:52.837962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.837944 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:52.851946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.850866 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp"] Oct 31 01:04:52.904536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.904487 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdffc\" (UniqueName: \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:52.904896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.904865 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:52.905028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.904993 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:52.938097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.936550 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:52.938097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:52.936693 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:04:53.009232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.008890 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.009232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.009039 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdffc\" (UniqueName: \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.009232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.009135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.029233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.022798 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.029233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.023178 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.062230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.061776 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdffc\" (UniqueName: \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\") pod \"maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.174271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.174179 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:53.506101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.506073 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp"] Oct 31 01:04:53.507217 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:53.507177 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074658a2_f4a3_46b7_b273_3fd2ac8ab17a.slice/crio-3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92 WatchSource:0}: Error finding container 3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92: Status 404 returned error can't find the container with id 3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92 Oct 31 01:04:53.516508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.516484 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq"] Oct 31 01:04:53.522025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.521999 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.558503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.540853 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq"] Oct 31 01:04:53.618332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.618300 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkt8m\" (UniqueName: \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.618469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.618380 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.618532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.618489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.720335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.720299 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkt8m\" (UniqueName: \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.720496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.720349 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.720566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.720494 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.721428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.721401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.724124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.724098 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.745608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.745588 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkt8m\" (UniqueName: \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\") pod \"maas-api-revoke-sub-048e64cf-hz9lq\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:53.854817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:53.854791 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:54.166512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.166482 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq"] Oct 31 01:04:54.166933 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:04:54.166906 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed25d9b_0eff_4779_8414_92228160f332.slice/crio-73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c WatchSource:0}: Error finding container 73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c: Status 404 returned error can't find the container with id 73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c Oct 31 01:04:54.504382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.504346 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" event={"ID":"074658a2-f4a3-46b7-b273-3fd2ac8ab17a","Type":"ContainerStarted","Data":"e9b456b05e84230a43afec543818f1ccb85b0875d4d30c784bcdb41ae700013a"} Oct 31 01:04:54.504555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.504439 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" event={"ID":"074658a2-f4a3-46b7-b273-3fd2ac8ab17a","Type":"ContainerStarted","Data":"3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92"} Oct 31 01:04:54.505934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.505905 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" event={"ID":"7ed25d9b-0eff-4779-8414-92228160f332","Type":"ContainerStarted","Data":"5105e4c277467a1e28a12c86a8a419611115ded455c9fd17e305919fd7879ef0"} Oct 31 01:04:54.506048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.505952 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" event={"ID":"7ed25d9b-0eff-4779-8414-92228160f332","Type":"ContainerStarted","Data":"73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c"} Oct 31 01:04:54.549617 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.549567 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" podStartSLOduration=2.549554564 podStartE2EDuration="2.549554564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:54.545700582 +0000 UTC m=+2762.994788234" watchObservedRunningTime="2026-10-31 01:04:54.549554564 +0000 UTC m=+2762.998642203" Oct 31 01:04:54.588377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:54.588317 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" podStartSLOduration=1.588301261 podStartE2EDuration="1.588301261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:54.585991956 +0000 UTC m=+2763.035079599" watchObservedRunningTime="2026-10-31 01:04:54.588301261 +0000 UTC m=+2763.037388900" Oct 31 01:04:55.515885 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:55.515803 5011 generic.go:356] "Generic (PLEG): container finished" podID="074658a2-f4a3-46b7-b273-3fd2ac8ab17a" containerID="e9b456b05e84230a43afec543818f1ccb85b0875d4d30c784bcdb41ae700013a" exitCode=0 Oct 31 01:04:55.516341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:55.515890 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" event={"ID":"074658a2-f4a3-46b7-b273-3fd2ac8ab17a","Type":"ContainerDied","Data":"e9b456b05e84230a43afec543818f1ccb85b0875d4d30c784bcdb41ae700013a"} Oct 31 01:04:55.517651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:55.517625 5011 generic.go:356] "Generic (PLEG): container finished" podID="7ed25d9b-0eff-4779-8414-92228160f332" containerID="5105e4c277467a1e28a12c86a8a419611115ded455c9fd17e305919fd7879ef0" exitCode=0 Oct 31 01:04:55.517784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:55.517661 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" event={"ID":"7ed25d9b-0eff-4779-8414-92228160f332","Type":"ContainerDied","Data":"5105e4c277467a1e28a12c86a8a419611115ded455c9fd17e305919fd7879ef0"} Oct 31 01:04:56.708094 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.708062 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:56.769666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.769191 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\") pod \"7ed25d9b-0eff-4779-8414-92228160f332\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " Oct 31 01:04:56.770662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.770638 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\" (UniqueName: \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\") pod \"7ed25d9b-0eff-4779-8414-92228160f332\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " Oct 31 01:04:56.771362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.771328 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\") pod \"7ed25d9b-0eff-4779-8414-92228160f332\" (UID: \"7ed25d9b-0eff-4779-8414-92228160f332\") " Oct 31 01:04:56.771971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.771708 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca" pod "7ed25d9b-0eff-4779-8414-92228160f332" (UID: "7ed25d9b-0eff-4779-8414-92228160f332"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:56.772072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.772036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca" pod "7ed25d9b-0eff-4779-8414-92228160f332" (UID: "7ed25d9b-0eff-4779-8414-92228160f332"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:56.772772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.772745 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:56.772885 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.772778 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed25d9b-0eff-4779-8414-92228160f332-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:56.775037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.775006 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m" pod "7ed25d9b-0eff-4779-8414-92228160f332" (UID: "7ed25d9b-0eff-4779-8414-92228160f332"). InnerVolumeSpecName "kube-api-access-kkt8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:56.776076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.776024 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:04:56.873470 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.873428 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\") pod \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " Oct 31 01:04:56.873654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.873477 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\") pod \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " Oct 31 01:04:56.873654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.873585 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\" (UniqueName: \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\") pod \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\" (UID: \"074658a2-f4a3-46b7-b273-3fd2ac8ab17a\") " Oct 31 01:04:56.873897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.873868 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca" pod "074658a2-f4a3-46b7-b273-3fd2ac8ab17a" (UID: "074658a2-f4a3-46b7-b273-3fd2ac8ab17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:56.874218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.874187 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:56.874288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.874225 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkt8m\" (UniqueName: \"kubernetes.io/projected/7ed25d9b-0eff-4779-8414-92228160f332-kube-api-access-kkt8m\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:56.875559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.875537 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca" pod "074658a2-f4a3-46b7-b273-3fd2ac8ab17a" (UID: "074658a2-f4a3-46b7-b273-3fd2ac8ab17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:56.875644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.875604 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc" pod "074658a2-f4a3-46b7-b273-3fd2ac8ab17a" (UID: "074658a2-f4a3-46b7-b273-3fd2ac8ab17a"). InnerVolumeSpecName "kube-api-access-sdffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:56.975741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.975707 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:56.975741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:56.975736 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdffc\" (UniqueName: \"kubernetes.io/projected/074658a2-f4a3-46b7-b273-3fd2ac8ab17a-kube-api-access-sdffc\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:04:57.536577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.536543 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" Oct 31 01:04:57.536789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.536581 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq" event={"ID":"7ed25d9b-0eff-4779-8414-92228160f332","Type":"ContainerDied","Data":"73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c"} Oct 31 01:04:57.536789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.536640 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a4cc7b65acc7198185501280d99ca8b89a44e8f3e7b2ff23811ce46358c74c" Oct 31 01:04:57.538332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.538304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" event={"ID":"074658a2-f4a3-46b7-b273-3fd2ac8ab17a","Type":"ContainerDied","Data":"3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92"} Oct 31 01:04:57.538469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.538349 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3731aaeabbdbe74d6f93622d96d1de36fe0b0ef1641eb1bbdaf67a1613edda92" Oct 31 01:04:57.538469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:04:57.538403 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp" Oct 31 01:05:00.256277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:00.256229 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 01:05:00.259146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:00.259107 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" containerID="cri-o://21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4" gracePeriod=30 Oct 31 01:05:01.949315 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:01.949279 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:05:04.684607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:04.684568 5011 patch_prober.go:29] interesting pod/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:05:04.685288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:04.685245 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:05:05.570997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.570963 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 01:05:05.631056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.631012 5011 generic.go:356] "Generic (PLEG): container finished" podID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerID="21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4" exitCode=0 Oct 31 01:05:05.631291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.631262 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" event={"ID":"76bc85df-4267-40d4-a5d6-dac43094d75e","Type":"ContainerDied","Data":"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4"} Oct 31 01:05:05.638110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.631330 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" event={"ID":"76bc85df-4267-40d4-a5d6-dac43094d75e","Type":"ContainerDied","Data":"73c81ee15546602bca435030e74b93c606d59408f291e72cf25a2f72146c1960"} Oct 31 01:05:05.638110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.631364 5011 scope.go:122] "RemoveContainer" containerID="21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4" Oct 31 01:05:05.638110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.631534 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv" Oct 31 01:05:05.647123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.647102 5011 scope.go:122] "RemoveContainer" containerID="21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4" Oct 31 01:05:05.647492 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:05.647456 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4\": container with ID starting with 21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4 not found: ID does not exist" containerID="21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4" Oct 31 01:05:05.647621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.647503 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4"} err="failed to get container status \"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4\": rpc error: code = NotFound desc = could not find container \"21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4\": container with ID starting with 21a9b537a5d03f48600c67cd283e9b08f37d2f2b1f80f7710962c41874dbace4 not found: ID does not exist" Oct 31 01:05:05.735882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.735855 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.735924 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.735988 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736044 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736073 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736114 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736138 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736297 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736342 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\") pod \"76bc85df-4267-40d4-a5d6-dac43094d75e\" (UID: \"76bc85df-4267-40d4-a5d6-dac43094d75e\") " Oct 31 01:05:05.736774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736344 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:05.736774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736376 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:05.736774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736511 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:05.736774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736748 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:05.737056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.736980 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.737056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.737000 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.737056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.737014 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.737056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.737029 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc85df-4267-40d4-a5d6-dac43094d75e-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.737056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.737043 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.738173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.738143 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:05.738484 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.738465 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "kube-api-access-7ws2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:05.738751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.738726 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:05:05.738861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.738748 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token" pod "76bc85df-4267-40d4-a5d6-dac43094d75e" (UID: "76bc85df-4267-40d4-a5d6-dac43094d75e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:05.838252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.838222 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.838252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.838249 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.838252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.838258 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ws2r\" (UniqueName: \"kubernetes.io/projected/76bc85df-4267-40d4-a5d6-dac43094d75e-kube-api-access-7ws2r\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.838573 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.838268 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc85df-4267-40d4-a5d6-dac43094d75e-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:05.982338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.982307 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 01:05:05.994259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:05.994232 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9de7f5-openshift-default-84c9b4bf8f-l5qkv"] Oct 31 01:05:06.960962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:06.960916 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 01:05:06.961511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:06.961191 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" containerID="cri-o://7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1" gracePeriod=30 Oct 31 01:05:07.685126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:07.685090 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" path="/var/lib/kubelet/pods/76bc85df-4267-40d4-a5d6-dac43094d75e/volumes" Oct 31 01:05:07.961053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:07.960974 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 01:05:07.961501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:07.961179 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" containerID="cri-o://15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499" gracePeriod=30 Oct 31 01:05:10.790630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:10.790595 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:05:10.790994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:10.790648 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:05:11.683397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:11.683307 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:05:11.683397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:11.683378 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:05:12.247536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.247506 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 01:05:12.323023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.322987 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323054 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323160 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323247 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323279 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323317 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323409 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323436 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323519 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\") pod \"daaf1adf-a248-461c-ab68-f7743527db00\" (UID: \"daaf1adf-a248-461c-ab68-f7743527db00\") " Oct 31 01:05:12.323548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323530 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:12.323884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.323657 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:12.324260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.324027 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:12.324260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.324124 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daaf1adf-a248-461c-ab68-f7743527db00-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.324260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.324150 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.324260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.324165 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.324656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.324599 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:12.327041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.327013 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:05:12.330636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.330612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:12.331361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.331333 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "kube-api-access-g8l9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:12.332315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.332294 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy" pod "daaf1adf-a248-461c-ab68-f7743527db00" (UID: "daaf1adf-a248-461c-ab68-f7743527db00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:12.425927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425892 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daaf1adf-a248-461c-ab68-f7743527db00-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.425927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425925 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.425927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425935 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8l9w\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-kube-api-access-g8l9w\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.426148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425944 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.426148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425953 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daaf1adf-a248-461c-ab68-f7743527db00-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.426148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.425962 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daaf1adf-a248-461c-ab68-f7743527db00-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:12.696643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.696608 5011 generic.go:356] "Generic (PLEG): container finished" podID="daaf1adf-a248-461c-ab68-f7743527db00" containerID="7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1" exitCode=0 Oct 31 01:05:12.696876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.696648 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" Oct 31 01:05:12.696876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.696691 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" event={"ID":"daaf1adf-a248-461c-ab68-f7743527db00","Type":"ContainerDied","Data":"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1"} Oct 31 01:05:12.696876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.696745 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g" event={"ID":"daaf1adf-a248-461c-ab68-f7743527db00","Type":"ContainerDied","Data":"0eb778f47ab310404d05312c90bcc026173708de4fc44aa516b7f13b7427ea82"} Oct 31 01:05:12.696876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.696778 5011 scope.go:122] "RemoveContainer" containerID="7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1" Oct 31 01:05:12.707986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.707970 5011 scope.go:122] "RemoveContainer" containerID="7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1" Oct 31 01:05:12.708256 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:12.708232 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1\": container with ID starting with 7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1 not found: ID does not exist" containerID="7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1" Oct 31 01:05:12.708323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.708265 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1"} err="failed to get container status \"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1\": rpc error: code = NotFound desc = could not find container \"7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1\": container with ID starting with 7a05ae87b1cc243a714b3b2abd57c7842eed098434085b680d45f1b72387a6f1 not found: ID does not exist" Oct 31 01:05:12.758950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.755899 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 01:05:12.762090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:12.762059 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xnm5g"] Oct 31 01:05:13.270296 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.270273 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 01:05:13.342296 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342263 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342319 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342430 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342486 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342624 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342668 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.342748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342727 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342751 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:13.342859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342830 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342923 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.342984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\") pod \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\" (UID: \"661ccd2b-c57d-4fc7-882d-e55eebbf2092\") " Oct 31 01:05:13.343098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.342979 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.343098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343067 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.343226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.343609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343578 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.343730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343616 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.343730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343630 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.343730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343645 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.343730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.343657 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.345759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.345736 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:13.345883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.345769 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:05:13.345883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.345798 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.346013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.345935 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj" pod "661ccd2b-c57d-4fc7-882d-e55eebbf2092" (UID: "661ccd2b-c57d-4fc7-882d-e55eebbf2092"). InnerVolumeSpecName "kube-api-access-rnscj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:13.444952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.444919 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-kube-api-access-rnscj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.444952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.444949 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.444952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.444960 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.445166 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.444972 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661ccd2b-c57d-4fc7-882d-e55eebbf2092-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.685002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.684960 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daaf1adf-a248-461c-ab68-f7743527db00" path="/var/lib/kubelet/pods/daaf1adf-a248-461c-ab68-f7743527db00/volumes" Oct 31 01:05:13.707461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.707428 5011 generic.go:356] "Generic (PLEG): container finished" podID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerID="15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499" exitCode=0 Oct 31 01:05:13.707604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.707489 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" Oct 31 01:05:13.707604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.707544 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" event={"ID":"661ccd2b-c57d-4fc7-882d-e55eebbf2092","Type":"ContainerDied","Data":"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499"} Oct 31 01:05:13.707604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.707590 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m" event={"ID":"661ccd2b-c57d-4fc7-882d-e55eebbf2092","Type":"ContainerDied","Data":"81c39a7c8646c71823495411b2201ffc733596e4dfdfe15328489dff30e53281"} Oct 31 01:05:13.707753 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.707618 5011 scope.go:122] "RemoveContainer" containerID="15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499" Oct 31 01:05:13.719774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.719742 5011 scope.go:122] "RemoveContainer" containerID="15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499" Oct 31 01:05:13.720056 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:13.720037 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499\": container with ID starting with 15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499 not found: ID does not exist" containerID="15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499" Oct 31 01:05:13.720118 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.720064 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499"} err="failed to get container status \"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499\": rpc error: code = NotFound desc = could not find container \"15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499\": container with ID starting with 15eab96ac7e2ace9b134ea57a2615f49dec15d4a4f6ad23f46f0d26da0072499 not found: ID does not exist" Oct 31 01:05:13.767730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.767701 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 01:05:13.777857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:13.777833 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-zpj5m"] Oct 31 01:05:14.709334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.709301 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw"] Oct 31 01:05:14.710259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710234 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7ed25d9b-0eff-4779-8414-92228160f332" containerName="revoke-keys" Oct 31 01:05:14.710259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710258 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed25d9b-0eff-4779-8414-92228160f332" containerName="revoke-keys" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710287 5011 cpu_manager.go:414] "containerMap: removing container" podUID="074658a2-f4a3-46b7-b273-3fd2ac8ab17a" containerName="revoke-keys" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710295 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074658a2-f4a3-46b7-b273-3fd2ac8ab17a" containerName="revoke-keys" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710334 5011 cpu_manager.go:414] "containerMap: removing container" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710342 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710376 5011 cpu_manager.go:414] "containerMap: removing container" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710383 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710419 5011 cpu_manager.go:414] "containerMap: removing container" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" Oct 31 01:05:14.710433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710426 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" Oct 31 01:05:14.710734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710599 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaf1adf-a248-461c-ab68-f7743527db00" containerName="istio-proxy" Oct 31 01:05:14.710734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710615 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc85df-4267-40d4-a5d6-dac43094d75e" containerName="istio-proxy" Oct 31 01:05:14.710734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710631 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" containerName="istio-proxy" Oct 31 01:05:14.710734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710639 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed25d9b-0eff-4779-8414-92228160f332" containerName="revoke-keys" Oct 31 01:05:14.710734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.710651 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="074658a2-f4a3-46b7-b273-3fd2ac8ab17a" containerName="revoke-keys" Oct 31 01:05:14.716418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.716395 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.734631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.734603 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw"] Oct 31 01:05:14.740582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.740553 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:14.740717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.740653 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:05:14.767634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.767602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbv7\" (UniqueName: \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.767782 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.767729 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.767841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.767809 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.868779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.868740 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbv7\" (UniqueName: \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.868973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.868801 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.868973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.868849 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.869569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.869544 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.871436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.871410 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:14.892952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:14.892928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbv7\" (UniqueName: \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\") pod \"maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:15.038085 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:15.038003 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:15.349442 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:15.349403 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw"] Oct 31 01:05:15.683458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:15.683425 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661ccd2b-c57d-4fc7-882d-e55eebbf2092" path="/var/lib/kubelet/pods/661ccd2b-c57d-4fc7-882d-e55eebbf2092/volumes" Oct 31 01:05:15.727009 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:15.726977 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" event={"ID":"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a","Type":"ContainerStarted","Data":"651d003e544b7d92a1329c4d1136d7a9a4079188de94c3fc7c4664161c8c8202"} Oct 31 01:05:16.735382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:16.735305 5011 generic.go:356] "Generic (PLEG): container finished" podID="78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" containerID="8308ab8c6dbc4b261a3649305ff50634d3c7db8fe12f010d7e33f27d39cd8055" exitCode=0 Oct 31 01:05:16.735765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:16.735389 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" event={"ID":"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a","Type":"ContainerDied","Data":"8308ab8c6dbc4b261a3649305ff50634d3c7db8fe12f010d7e33f27d39cd8055"} Oct 31 01:05:17.032067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:17.031990 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd"] Oct 31 01:05:17.035451 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:17.035427 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af490a1-nc9kd"] Oct 31 01:05:17.683415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:17.683378 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad99619e-203e-41b8-a029-6b434053aae1" path="/var/lib/kubelet/pods/ad99619e-203e-41b8-a029-6b434053aae1/volumes" Oct 31 01:05:17.916001 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:17.915978 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:18.008716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.008644 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\") pod \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " Oct 31 01:05:18.008716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.008710 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\") pod \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " Oct 31 01:05:18.008921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.008888 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\" (UniqueName: \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\") pod \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\" (UID: \"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a\") " Oct 31 01:05:18.009008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.008983 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca" pod "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" (UID: "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:18.009459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.009427 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:18.010879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.010849 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca" pod "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" (UID: "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:18.010986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.010940 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7" pod "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" (UID: "78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a"). InnerVolumeSpecName "kube-api-access-zcbv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:18.112707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.112672 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbv7\" (UniqueName: \"kubernetes.io/projected/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-kube-api-access-zcbv7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:18.112707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.112706 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:18.753218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.753177 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" Oct 31 01:05:18.753399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.753172 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw" event={"ID":"78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a","Type":"ContainerDied","Data":"651d003e544b7d92a1329c4d1136d7a9a4079188de94c3fc7c4664161c8c8202"} Oct 31 01:05:18.753399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:18.753312 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651d003e544b7d92a1329c4d1136d7a9a4079188de94c3fc7c4664161c8c8202" Oct 31 01:05:23.887202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.887166 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:05:23.888024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.887999 5011 cpu_manager.go:414] "containerMap: removing container" podUID="78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" containerName="revoke-keys" Oct 31 01:05:23.888070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.888029 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" containerName="revoke-keys" Oct 31 01:05:23.888313 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.888290 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" containerName="revoke-keys" Oct 31 01:05:23.892100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.892079 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.917157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.917125 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:05:23.920808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.920783 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"openshift-service-ca.crt\"" Oct 31 01:05:23.921152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.921119 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"auth-test-model-498580-kserve-self-signed-certs\"" Oct 31 01:05:23.921469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.921447 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"kube-root-ca.crt\"" Oct 31 01:05:23.921694 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.921671 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"default-dockercfg-ll2nn\"" Oct 31 01:05:23.993662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.993626 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.993814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.993693 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.993814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.993770 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.993897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.993841 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.993897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.993879 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkcgv\" (UniqueName: \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:23.994053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:23.994029 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.094953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095006 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkcgv\" (UniqueName: \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095121 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095305 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095454 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095745 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.095774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.095761 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.098655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.098627 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.124428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.122765 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.124428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.123371 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkcgv\" (UniqueName: \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\") pod \"auth-test-model-498580-kserve-5cdd747ccc-jn7kq\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.231837 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.231462 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:24.609637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.609604 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:05:24.610430 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:05:24.610405 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6edc1a0_4145_45ec_9372_f3fa9f52f045.slice/crio-3680cc442eeb4581f13b3ae3fa8ec56bac324dd111e80cd4fd6d19cfc18dbf2b WatchSource:0}: Error finding container 3680cc442eeb4581f13b3ae3fa8ec56bac324dd111e80cd4fd6d19cfc18dbf2b: Status 404 returned error can't find the container with id 3680cc442eeb4581f13b3ae3fa8ec56bac324dd111e80cd4fd6d19cfc18dbf2b Oct 31 01:05:24.797498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.797461 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" event={"ID":"d6edc1a0-4145-45ec-9372-f3fa9f52f045","Type":"ContainerStarted","Data":"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406"} Oct 31 01:05:24.797674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.797514 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" event={"ID":"d6edc1a0-4145-45ec-9372-f3fa9f52f045","Type":"ContainerStarted","Data":"3680cc442eeb4581f13b3ae3fa8ec56bac324dd111e80cd4fd6d19cfc18dbf2b"} Oct 31 01:05:24.842881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:24.842793 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" podStartSLOduration=1.842775289 podStartE2EDuration="1.842775289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:24.838236565 +0000 UTC m=+2793.287324203" watchObservedRunningTime="2026-10-31 01:05:24.842775289 +0000 UTC m=+2793.291862928" Oct 31 01:05:25.232699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:25.232668 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:30.064055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.064014 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 01:05:30.064846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.064806 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" podUID="38f0ca78-8042-4926-8e87-69510deea04d" containerName="maas-api" containerID="cri-o://c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b" gracePeriod=30 Oct 31 01:05:30.629803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.629727 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 01:05:30.719070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.719027 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\" (UniqueName: \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\") pod \"38f0ca78-8042-4926-8e87-69510deea04d\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " Oct 31 01:05:30.719288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.719097 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\") pod \"38f0ca78-8042-4926-8e87-69510deea04d\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " Oct 31 01:05:30.719288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.719197 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\") pod \"38f0ca78-8042-4926-8e87-69510deea04d\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " Oct 31 01:05:30.719424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.719301 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\") pod \"38f0ca78-8042-4926-8e87-69510deea04d\" (UID: \"38f0ca78-8042-4926-8e87-69510deea04d\") " Oct 31 01:05:30.720597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.720571 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca" pod "38f0ca78-8042-4926-8e87-69510deea04d" (UID: "38f0ca78-8042-4926-8e87-69510deea04d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:30.723677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.723650 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs" pod "38f0ca78-8042-4926-8e87-69510deea04d" (UID: "38f0ca78-8042-4926-8e87-69510deea04d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:30.731245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.731199 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls" pod "38f0ca78-8042-4926-8e87-69510deea04d" (UID: "38f0ca78-8042-4926-8e87-69510deea04d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:30.735067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.735038 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz" pod "38f0ca78-8042-4926-8e87-69510deea04d" (UID: "38f0ca78-8042-4926-8e87-69510deea04d"). InnerVolumeSpecName "kube-api-access-xwcbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:30.821453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.821423 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:30.821453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.821452 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f0ca78-8042-4926-8e87-69510deea04d-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:30.821624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.821463 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcbz\" (UniqueName: \"kubernetes.io/projected/38f0ca78-8042-4926-8e87-69510deea04d-kube-api-access-xwcbz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:30.821624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.821472 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f0ca78-8042-4926-8e87-69510deea04d-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:30.838896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.838868 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp"] Oct 31 01:05:30.842625 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.842600 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-01a117-n9gfp"] Oct 31 01:05:30.848631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.848606 5011 generic.go:356] "Generic (PLEG): container finished" podID="38f0ca78-8042-4926-8e87-69510deea04d" containerID="c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b" exitCode=0 Oct 31 01:05:30.848727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.848653 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" Oct 31 01:05:30.848727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.848692 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" event={"ID":"38f0ca78-8042-4926-8e87-69510deea04d","Type":"ContainerDied","Data":"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b"} Oct 31 01:05:30.848802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.848746 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw" event={"ID":"38f0ca78-8042-4926-8e87-69510deea04d","Type":"ContainerDied","Data":"1a6e5f8df50d85d1f48b65d70ef15aba28c202854ee67c9e6707a776564092d7"} Oct 31 01:05:30.848802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.848776 5011 scope.go:122] "RemoveContainer" containerID="c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b" Oct 31 01:05:30.860711 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.860691 5011 scope.go:122] "RemoveContainer" containerID="c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b" Oct 31 01:05:30.860991 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:30.860971 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b\": container with ID starting with c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b not found: ID does not exist" containerID="c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b" Oct 31 01:05:30.861033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.861001 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b"} err="failed to get container status \"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b\": rpc error: code = NotFound desc = could not find container \"c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b\": container with ID starting with c1b80d3cf0aff37d96824973f88d2600527b44ac100aadabeb4944099a77a53b not found: ID does not exist" Oct 31 01:05:30.904082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.904008 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 01:05:30.908780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:30.908751 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01a117-8d45bf946-xhfmw"] Oct 31 01:05:31.684675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:31.684643 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074658a2-f4a3-46b7-b273-3fd2ac8ab17a" path="/var/lib/kubelet/pods/074658a2-f4a3-46b7-b273-3fd2ac8ab17a/volumes" Oct 31 01:05:31.685178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:31.685160 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f0ca78-8042-4926-8e87-69510deea04d" path="/var/lib/kubelet/pods/38f0ca78-8042-4926-8e87-69510deea04d/volumes" Oct 31 01:05:31.949282 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:31.948548 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:05:33.660188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.660151 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:05:33.661871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.661846 5011 cpu_manager.go:414] "containerMap: removing container" podUID="38f0ca78-8042-4926-8e87-69510deea04d" containerName="maas-api" Oct 31 01:05:33.662049 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.662033 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f0ca78-8042-4926-8e87-69510deea04d" containerName="maas-api" Oct 31 01:05:33.662359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.662340 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f0ca78-8042-4926-8e87-69510deea04d" containerName="maas-api" Oct 31 01:05:33.667449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.667426 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.696668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.696636 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-476a67\"/\"openshift-service-ca.crt\"" Oct 31 01:05:33.696818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.696747 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-476a67\"/\"kube-root-ca.crt\"" Oct 31 01:05:33.699292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.699261 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:05:33.708808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.708787 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-476a67\"/\"default-dockercfg-4xz4d\"" Oct 31 01:05:33.708911 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.708897 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-476a67\"/\"auth-test-model-476a67-kserve-self-signed-certs\"" Oct 31 01:05:33.760406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.760370 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.760666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.760645 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.760861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.760838 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.761143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.761118 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbj7\" (UniqueName: \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.761530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.761504 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.761642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.761549 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862402 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbj7\" (UniqueName: \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862476 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862570 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862663 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.862814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.862713 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.863169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.863138 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.863467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.863446 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.863701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.863681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.865887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.865862 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.866112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.866091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.911280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.901576 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbj7\" (UniqueName: \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\") pod \"auth-test-model-476a67-kserve-65c76885d9-hc9tk\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:33.992532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:33.992494 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:34.233430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.233348 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:34.246542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.246513 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:34.353889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.353857 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:05:34.354013 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:05:34.353929 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb05376bb_68b1_4ace_9729_3d3b557ee305.slice/crio-ddeba1009cc600f582d4cddd28ddae600d815c28e23e0974856d5ac417e98c49 WatchSource:0}: Error finding container ddeba1009cc600f582d4cddd28ddae600d815c28e23e0974856d5ac417e98c49: Status 404 returned error can't find the container with id ddeba1009cc600f582d4cddd28ddae600d815c28e23e0974856d5ac417e98c49 Oct 31 01:05:34.887110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.887072 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" event={"ID":"b05376bb-68b1-4ace-9729-3d3b557ee305","Type":"ContainerStarted","Data":"c8babab1d9d95a141e9c26cf0c44c8ef4be428ef5638b4095b6dc87e20cf8a89"} Oct 31 01:05:34.887519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.887123 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" event={"ID":"b05376bb-68b1-4ace-9729-3d3b557ee305","Type":"ContainerStarted","Data":"ddeba1009cc600f582d4cddd28ddae600d815c28e23e0974856d5ac417e98c49"} Oct 31 01:05:34.898567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.898537 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:05:34.929959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.929912 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" podStartSLOduration=1.929899928 podStartE2EDuration="1.929899928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:34.927674076 +0000 UTC m=+2803.376761715" watchObservedRunningTime="2026-10-31 01:05:34.929899928 +0000 UTC m=+2803.378987588" Oct 31 01:05:34.994127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:34.994096 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:36.085779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:36.085743 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:05:38.577141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:38.577106 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw"] Oct 31 01:05:38.588806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:38.588776 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72120c-v8pzw"] Oct 31 01:05:39.769627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:39.769588 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a" path="/var/lib/kubelet/pods/78ef0e4d-b727-4d94-ad2d-3e25dcd03f0a/volumes" Oct 31 01:05:43.993622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:43.993585 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:44.011676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:44.011647 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:44.060821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:44.060788 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:05:46.214075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.214035 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 01:05:46.214645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.214364 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" containerID="cri-o://d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380" gracePeriod=30 Oct 31 01:05:46.823011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.821038 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l"] Oct 31 01:05:46.831023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.830994 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:46.855197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.855156 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l"] Oct 31 01:05:46.907232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.907173 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:46.907527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.907489 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:05:46.915890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.915862 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxdz\" (UniqueName: \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:46.916012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.915925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:46.916073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:46.916004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.017635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.017592 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.017840 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.017649 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.017973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.017948 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxdz\" (UniqueName: \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.019290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.019262 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.022194 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.022163 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.050353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.045058 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxdz\" (UniqueName: \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\") pod \"maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.172151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.172069 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:47.531173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:47.531140 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l"] Oct 31 01:05:48.073820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.073774 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" event={"ID":"2b79504e-b7fb-4fc4-b9e5-c15b64f99610","Type":"ContainerStarted","Data":"3e0685da1fb5900df3242b5e4480e2a3ee1825cb2be46c81d7639c597e116f10"} Oct 31 01:05:48.074132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.073844 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" event={"ID":"2b79504e-b7fb-4fc4-b9e5-c15b64f99610","Type":"ContainerStarted","Data":"f6f1d807af4fa85b52f82dc3fa737c02cf1a00cd4201858014669d7ebc22ff55"} Oct 31 01:05:48.075904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.075878 5011 generic.go:356] "Generic (PLEG): container finished" podID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerID="ab09e36446414bedbe1f0b4568de590f69c1ae8f032a69e09ec16ac8d3f6719c" exitCode=0 Oct 31 01:05:48.076026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.075951 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerDied","Data":"ab09e36446414bedbe1f0b4568de590f69c1ae8f032a69e09ec16ac8d3f6719c"} Oct 31 01:05:48.076430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.076409 5011 scope.go:122] "RemoveContainer" containerID="ab09e36446414bedbe1f0b4568de590f69c1ae8f032a69e09ec16ac8d3f6719c" Oct 31 01:05:48.078025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.077999 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:48.078104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.078065 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:48.078104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.078097 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:48.120677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.120612 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" podStartSLOduration=2.120592918 podStartE2EDuration="2.120592918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:48.115409506 +0000 UTC m=+2816.564497147" watchObservedRunningTime="2026-10-31 01:05:48.120592918 +0000 UTC m=+2816.569680558" Oct 31 01:05:48.420470 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.420370 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:05:48.420470 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:48.420436 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:05:49.085803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.085762 5011 generic.go:356] "Generic (PLEG): container finished" podID="2b79504e-b7fb-4fc4-b9e5-c15b64f99610" containerID="3e0685da1fb5900df3242b5e4480e2a3ee1825cb2be46c81d7639c597e116f10" exitCode=0 Oct 31 01:05:49.086388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.085856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" event={"ID":"2b79504e-b7fb-4fc4-b9e5-c15b64f99610","Type":"ContainerDied","Data":"3e0685da1fb5900df3242b5e4480e2a3ee1825cb2be46c81d7639c597e116f10"} Oct 31 01:05:49.087572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.087549 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerID="504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074" exitCode=0 Oct 31 01:05:49.087711 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.087626 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerDied","Data":"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074"} Oct 31 01:05:49.088104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.088083 5011 scope.go:122] "RemoveContainer" containerID="504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074" Oct 31 01:05:49.090033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.090010 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerStarted","Data":"06d7c76a75cc9546c064ab10196acede9916adf8124041b397660ea94408fa6a"} Oct 31 01:05:49.090370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.090342 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:49.090476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.090428 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:49.090542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.090479 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:49.111797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.111776 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:05:49.112139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.112123 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:49.112261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:49.112162 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:50.101652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.101613 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerStarted","Data":"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48"} Oct 31 01:05:50.116620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.116302 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:50.116620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.116355 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:50.402225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.399868 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:50.457467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.457435 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:05:50.476005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.475807 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:50.476471 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.476269 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:50.506173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.506137 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\" (UniqueName: \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\") pod \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " Oct 31 01:05:50.506483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.506459 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\") pod \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " Oct 31 01:05:50.506653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.506633 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\") pod \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\" (UID: \"2b79504e-b7fb-4fc4-b9e5-c15b64f99610\") " Oct 31 01:05:50.508519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.508492 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca" pod "2b79504e-b7fb-4fc4-b9e5-c15b64f99610" (UID: "2b79504e-b7fb-4fc4-b9e5-c15b64f99610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:50.510394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.510370 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca" pod "2b79504e-b7fb-4fc4-b9e5-c15b64f99610" (UID: "2b79504e-b7fb-4fc4-b9e5-c15b64f99610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:50.510612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.510584 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz" pod "2b79504e-b7fb-4fc4-b9e5-c15b64f99610" (UID: "2b79504e-b7fb-4fc4-b9e5-c15b64f99610"). InnerVolumeSpecName "kube-api-access-gwxdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:50.609424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.609393 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxdz\" (UniqueName: \"kubernetes.io/projected/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-kube-api-access-gwxdz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:50.609424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.609420 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:50.609424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:50.609432 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b79504e-b7fb-4fc4-b9e5-c15b64f99610-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:51.111891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.111848 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:51.112327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.111916 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:51.147234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.128918 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" Oct 31 01:05:51.148221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.148174 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l" event={"ID":"2b79504e-b7fb-4fc4-b9e5-c15b64f99610","Type":"ContainerDied","Data":"f6f1d807af4fa85b52f82dc3fa737c02cf1a00cd4201858014669d7ebc22ff55"} Oct 31 01:05:51.148452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.148433 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f1d807af4fa85b52f82dc3fa737c02cf1a00cd4201858014669d7ebc22ff55" Oct 31 01:05:51.466232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.462512 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:51.466232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.462585 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:51.729060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.728526 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 01:05:51.900768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.900727 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901035 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901235 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901375 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901523 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901683 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.901851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.901834 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.902043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.902028 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.906240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.903441 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\") pod \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\" (UID: \"8016e34f-8e5f-4f34-865e-ce8b044c3ff6\") " Oct 31 01:05:51.906240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.902700 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:51.908185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.907195 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:51.908185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.907467 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:51.908185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.907637 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:51.908185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.907798 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:51.915125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.915096 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:51.915697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.915673 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "kube-api-access-ct9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:51.916147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.916124 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:05:51.916505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:51.916482 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token" pod "8016e34f-8e5f-4f34-865e-ce8b044c3ff6" (UID: "8016e34f-8e5f-4f34-865e-ce8b044c3ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:52.006138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006097 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006126 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006135 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9g7\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-kube-api-access-ct9g7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006144 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006154 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006162 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006175 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006183 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.006489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.006191 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8016e34f-8e5f-4f34-865e-ce8b044c3ff6-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:05:52.112792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.112754 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:52.113282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.112806 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:52.185228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.183724 5011 generic.go:356] "Generic (PLEG): container finished" podID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerID="d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380" exitCode=0 Oct 31 01:05:52.185228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.183923 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerDied","Data":"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380"} Oct 31 01:05:52.185228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.183974 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" event={"ID":"8016e34f-8e5f-4f34-865e-ce8b044c3ff6","Type":"ContainerDied","Data":"01a4602826c898743d7007876c1e616b56d4392d4228c700f86061780bb95ce5"} Oct 31 01:05:52.185228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.184003 5011 scope.go:122] "RemoveContainer" containerID="d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380" Oct 31 01:05:52.185228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.184164 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg" Oct 31 01:05:52.204116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.203176 5011 scope.go:122] "RemoveContainer" containerID="146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945" Oct 31 01:05:52.213618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.213591 5011 scope.go:122] "RemoveContainer" containerID="d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380" Oct 31 01:05:52.213910 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:52.213889 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380\": container with ID starting with d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380 not found: ID does not exist" containerID="d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380" Oct 31 01:05:52.213985 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.213917 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380"} err="failed to get container status \"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380\": rpc error: code = NotFound desc = could not find container \"d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380\": container with ID starting with d3f0a4b9ab316abe3bdd9834b501290aad7c6ce95e526680ca0e87bb3e807380 not found: ID does not exist" Oct 31 01:05:52.213985 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.213934 5011 scope.go:122] "RemoveContainer" containerID="146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945" Oct 31 01:05:52.214148 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:05:52.214127 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945\": container with ID starting with 146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945 not found: ID does not exist" containerID="146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945" Oct 31 01:05:52.214193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.214155 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945"} err="failed to get container status \"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945\": rpc error: code = NotFound desc = could not find container \"146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945\": container with ID starting with 146612b74dcb280d234bc60c92ed483866bbb8fe53857c994e462234fc56c945 not found: ID does not exist" Oct 31 01:05:52.284406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.284290 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 01:05:52.287970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.287559 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72120c-openshift-default-77d78ddc45-gh6gg"] Oct 31 01:05:52.472281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.470345 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:52.472281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:52.470417 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:53.114181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:53.113533 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:53.114181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:53.113593 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:53.461648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:53.461555 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:53.462232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:53.461866 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:53.725236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:53.718683 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" path="/var/lib/kubelet/pods/8016e34f-8e5f-4f34-865e-ce8b044c3ff6/volumes" Oct 31 01:05:54.117636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:54.117593 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:05:54.120697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:54.120657 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 01:05:54.457035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:54.456947 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:54.461353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:54.461318 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:55.463229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:55.462651 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:05:55.463229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:55.462718 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 01:05:55.571231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:55.570480 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:05:55.573255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:55.572037 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:05:55.575667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:55.575643 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:05:56.462881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:56.462853 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:05:56.463427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:56.463405 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:05:56.464129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:05:56.464106 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:06:01.949382 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:06:01.949330 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:06:02.224596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:02.224500 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 01:06:02.224857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:02.224827 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" containerID="cri-o://bf3a271d0867029ba596f6edbc46813245f75b4747ba3e43cbdea2437ef15070" gracePeriod=30 Oct 31 01:06:02.234193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:02.234167 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 01:06:02.234392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:02.234371 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" containerID="cri-o://e115177fbaabc9ff21cc80fd9a9ec7c6c0d8c5800766203b9b761882eb89eaed" gracePeriod=30 Oct 31 01:06:03.684399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.684308 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw"] Oct 31 01:06:03.684969 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.684948 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.684971 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.684983 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.684989 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.685002 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2b79504e-b7fb-4fc4-b9e5-c15b64f99610" containerName="revoke-keys" Oct 31 01:06:03.685037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.685008 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b79504e-b7fb-4fc4-b9e5-c15b64f99610" containerName="revoke-keys" Oct 31 01:06:03.685328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.685099 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.685107 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8016e34f-8e5f-4f34-865e-ce8b044c3ff6" containerName="istio-proxy" Oct 31 01:06:03.685328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.685114 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b79504e-b7fb-4fc4-b9e5-c15b64f99610" containerName="revoke-keys" Oct 31 01:06:03.690313 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.690293 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.708947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.708923 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw"] Oct 31 01:06:03.717359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.717333 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:03.717807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.717756 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:06:03.792422 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.792385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.792595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.792538 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.792686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.792671 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsbw\" (UniqueName: \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.894364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.894334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.894533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.894411 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.894533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.894514 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsbw\" (UniqueName: \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.895061 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.895033 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.896846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.896821 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:03.918765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:03.918730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsbw\" (UniqueName: \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\") pod \"maas-api-revoke-sub-8b2ba884-w4vvw\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:04.004732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:04.004652 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:04.317047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:04.317014 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw"] Oct 31 01:06:04.317297 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:04.317267 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78017d7_f4c0_4e30_a319_ca7e8bd5ae04.slice/crio-6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f WatchSource:0}: Error finding container 6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f: Status 404 returned error can't find the container with id 6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f Oct 31 01:06:04.349949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:04.349915 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" event={"ID":"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04","Type":"ContainerStarted","Data":"6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f"} Oct 31 01:06:05.360604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.360570 5011 generic.go:356] "Generic (PLEG): container finished" podID="f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" containerID="8eeaeb93d1aa5967091cb20fe309b829818ee2ee832a4b796361dcc48bce5582" exitCode=0 Oct 31 01:06:05.360952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.360657 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" event={"ID":"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04","Type":"ContainerDied","Data":"8eeaeb93d1aa5967091cb20fe309b829818ee2ee832a4b796361dcc48bce5582"} Oct 31 01:06:05.524191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.524110 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vlrk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:06:05.524191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.524166 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:06:05.619737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.619703 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c7jth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:06:05.619910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:05.619760 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:06:06.539985 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.539954 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:06.647590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.647507 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\" (UniqueName: \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\") pod \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " Oct 31 01:06:06.647765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.647595 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\") pod \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " Oct 31 01:06:06.647765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.647648 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\") pod \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\" (UID: \"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04\") " Oct 31 01:06:06.647963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.647932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca" pod "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" (UID: "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:06.648316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.648295 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:06.649732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.649700 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw" pod "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" (UID: "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04"). InnerVolumeSpecName "kube-api-access-jtsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:06.649832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.649770 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca" pod "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" (UID: "f78017d7-f4c0-4e30-a319-ca7e8bd5ae04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:06.749916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.749889 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:06.749916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:06.749915 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsbw\" (UniqueName: \"kubernetes.io/projected/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04-kube-api-access-jtsbw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.382310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.382253 5011 generic.go:356] "Generic (PLEG): container finished" podID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerID="e115177fbaabc9ff21cc80fd9a9ec7c6c0d8c5800766203b9b761882eb89eaed" exitCode=0 Oct 31 01:06:07.382477 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.382423 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" event={"ID":"9df53695-6e13-49d5-bc1f-f8fe786df192","Type":"ContainerDied","Data":"e115177fbaabc9ff21cc80fd9a9ec7c6c0d8c5800766203b9b761882eb89eaed"} Oct 31 01:06:07.384045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.384009 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" event={"ID":"f78017d7-f4c0-4e30-a319-ca7e8bd5ae04","Type":"ContainerDied","Data":"6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f"} Oct 31 01:06:07.384233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.384056 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6744ec1e912a02254d533cb644c48ac900937ab64e9835db285bc48c69e2fd0f" Oct 31 01:06:07.385297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.385268 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw" Oct 31 01:06:07.386501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.386475 5011 generic.go:356] "Generic (PLEG): container finished" podID="bfcb33eb-927a-447d-9a30-cd024560647f" containerID="bf3a271d0867029ba596f6edbc46813245f75b4747ba3e43cbdea2437ef15070" exitCode=0 Oct 31 01:06:07.386614 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.386498 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" event={"ID":"bfcb33eb-927a-447d-9a30-cd024560647f","Type":"ContainerDied","Data":"bf3a271d0867029ba596f6edbc46813245f75b4747ba3e43cbdea2437ef15070"} Oct 31 01:06:07.546856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.546828 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 01:06:07.597707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.597679 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 01:06:07.665781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665750 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.665931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.665931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665840 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.665931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665871 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.665931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665917 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665958 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.665992 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.666181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666077 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666106 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666141 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666246 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666286 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666331 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666367 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666438 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666508 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.666884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666598 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\") pod \"9df53695-6e13-49d5-bc1f-f8fe786df192\" (UID: \"9df53695-6e13-49d5-bc1f-f8fe786df192\") " Oct 31 01:06:07.666884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.666643 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\") pod \"bfcb33eb-927a-447d-9a30-cd024560647f\" (UID: \"bfcb33eb-927a-447d-9a30-cd024560647f\") " Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.667866 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668243 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668510 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668534 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.668836 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668710 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.669358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.668962 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:07.669358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.669190 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.669517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.669391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.669691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.669667 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.670303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.670157 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:07.670303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.670247 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:07.670890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.670865 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:07.670986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.670949 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.671329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.671303 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "kube-api-access-cl58g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:07.671437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.671310 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:07.672013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.671988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token" pod "9df53695-6e13-49d5-bc1f-f8fe786df192" (UID: "9df53695-6e13-49d5-bc1f-f8fe786df192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:07.672114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.672045 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl" pod "bfcb33eb-927a-447d-9a30-cd024560647f" (UID: "bfcb33eb-927a-447d-9a30-cd024560647f"). InnerVolumeSpecName "kube-api-access-x4vbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:07.768781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768703 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vbl\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-kube-api-access-x4vbl\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.768781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768740 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.768781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768754 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.768781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768767 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.768781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768779 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcb33eb-927a-447d-9a30-cd024560647f-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768790 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcb33eb-927a-447d-9a30-cd024560647f-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768804 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcb33eb-927a-447d-9a30-cd024560647f-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768816 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768826 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768834 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768841 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768849 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768856 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768864 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768873 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcb33eb-927a-447d-9a30-cd024560647f-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768887 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df53695-6e13-49d5-bc1f-f8fe786df192-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768898 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df53695-6e13-49d5-bc1f-f8fe786df192-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:07.769057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:07.768906 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl58g\" (UniqueName: \"kubernetes.io/projected/9df53695-6e13-49d5-bc1f-f8fe786df192-kube-api-access-cl58g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:08.237026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.236989 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 01:06:08.237256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.237199 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" containerID="cri-o://046b85f6c9a41b548aec02bd600502e1116c0228641037e4dc085e564b17598d" gracePeriod=30 Oct 31 01:06:08.396236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.396170 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" Oct 31 01:06:08.396236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.396191 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth" event={"ID":"9df53695-6e13-49d5-bc1f-f8fe786df192","Type":"ContainerDied","Data":"18e7d112f8ae1c6c2ef4ee86f8d0dbea6c7ab60d1a5a8211e9209dc52a06c817"} Oct 31 01:06:08.396497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.396268 5011 scope.go:122] "RemoveContainer" containerID="e115177fbaabc9ff21cc80fd9a9ec7c6c0d8c5800766203b9b761882eb89eaed" Oct 31 01:06:08.397941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.397915 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" Oct 31 01:06:08.398076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.397968 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2" event={"ID":"bfcb33eb-927a-447d-9a30-cd024560647f","Type":"ContainerDied","Data":"40bcf1ee2b2ad87efee98708ce61b6531e82462bcb16bca52c4be4dce605cce4"} Oct 31 01:06:08.414130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.414110 5011 scope.go:122] "RemoveContainer" containerID="bf3a271d0867029ba596f6edbc46813245f75b4747ba3e43cbdea2437ef15070" Oct 31 01:06:08.450692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.450663 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 01:06:08.454361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.454339 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c7jth"] Oct 31 01:06:08.488570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.488512 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 01:06:08.495381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:08.495355 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vlrk2"] Oct 31 01:06:09.685121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:09.685081 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" path="/var/lib/kubelet/pods/9df53695-6e13-49d5-bc1f-f8fe786df192/volumes" Oct 31 01:06:09.685856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:09.685835 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" path="/var/lib/kubelet/pods/bfcb33eb-927a-447d-9a30-cd024560647f/volumes" Oct 31 01:06:11.513263 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:11.513227 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:06:11.513714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:11.513283 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:06:13.446911 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.446482 5011 generic.go:356] "Generic (PLEG): container finished" podID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerID="046b85f6c9a41b548aec02bd600502e1116c0228641037e4dc085e564b17598d" exitCode=0 Oct 31 01:06:13.446911 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.446771 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" event={"ID":"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5","Type":"ContainerDied","Data":"046b85f6c9a41b548aec02bd600502e1116c0228641037e4dc085e564b17598d"} Oct 31 01:06:13.511290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.511200 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 01:06:13.642970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.642937 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.642981 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643064 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643141 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643180 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643244 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:13.643324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643272 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643328 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:13.643404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643333 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643421 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643463 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:13.643584 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\") pod \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\" (UID: \"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5\") " Oct 31 01:06:13.643685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643660 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:13.643789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.643767 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:13.644196 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.644169 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.644331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.644201 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.644331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.644230 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.644331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.644245 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.644331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.644260 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.645591 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.645565 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:13.645712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.645594 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:13.645712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.645655 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:13.645879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.645846 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226" pod "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" (UID: "3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5"). InnerVolumeSpecName "kube-api-access-db226". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:13.745799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.745760 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db226\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-kube-api-access-db226\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.745799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.745792 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.745982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.745805 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:13.745982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:13.745818 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:14.456252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:14.456194 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" event={"ID":"3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5","Type":"ContainerDied","Data":"f8386ffa6c383948495685625705b81cfa93e6c5203cd1132f8c7dfae69822cf"} Oct 31 01:06:14.456902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:14.456280 5011 scope.go:122] "RemoveContainer" containerID="046b85f6c9a41b548aec02bd600502e1116c0228641037e4dc085e564b17598d" Oct 31 01:06:14.456902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:14.456311 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt" Oct 31 01:06:14.529429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:14.529398 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 01:06:14.541495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:14.541465 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-dq5tt"] Oct 31 01:06:15.683875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:15.683837 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" path="/var/lib/kubelet/pods/3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5/volumes" Oct 31 01:06:23.620999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.620960 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:06:23.621803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621779 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" Oct 31 01:06:23.621893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621806 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" Oct 31 01:06:23.621893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621827 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" containerName="revoke-keys" Oct 31 01:06:23.621893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621835 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" containerName="revoke-keys" Oct 31 01:06:23.621893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621855 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" Oct 31 01:06:23.621893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621864 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621896 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.621904 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.622085 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" containerName="revoke-keys" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.622099 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3803ca5d-c7cf-4aad-a5ef-debcce4ed7e5" containerName="istio-proxy" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.622112 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcb33eb-927a-447d-9a30-cd024560647f" containerName="istio-proxy" Oct 31 01:06:23.622182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.622130 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df53695-6e13-49d5-bc1f-f8fe786df192" containerName="istio-proxy" Oct 31 01:06:23.626162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.626139 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.648479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.648101 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:06:23.654436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.654411 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"kube-root-ca.crt\"" Oct 31 01:06:23.654749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.654727 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"default-dockercfg-td2h5\"" Oct 31 01:06:23.654931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.654913 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"openshift-service-ca.crt\"" Oct 31 01:06:23.658363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.658343 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"auth-test-model-3c90d8-kserve-self-signed-certs\"" Oct 31 01:06:23.664224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.664183 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74k7\" (UniqueName: \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.664461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.664440 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.664642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.664623 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.664939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.664921 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.665135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.665117 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.665417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.665397 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.766938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.766899 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74k7\" (UniqueName: \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.766953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.766984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767119 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767175 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767359 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767550 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767657 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.767745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.767717 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.769609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.769579 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.769914 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.769897 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.790571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.790542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74k7\" (UniqueName: \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\") pod \"auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:23.961278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:23.957849 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:24.314520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:24.314489 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:06:24.314809 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:24.314784 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91a4ffcd_25f8_4f6a_b399_6d50086a7978.slice/crio-0c31d930d2a2ac61f273ceeaa5a29ae6bb8a7711a456c4516cabd0de37b1397c WatchSource:0}: Error finding container 0c31d930d2a2ac61f273ceeaa5a29ae6bb8a7711a456c4516cabd0de37b1397c: Status 404 returned error can't find the container with id 0c31d930d2a2ac61f273ceeaa5a29ae6bb8a7711a456c4516cabd0de37b1397c Oct 31 01:06:24.534071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:24.534022 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" event={"ID":"91a4ffcd-25f8-4f6a-b399-6d50086a7978","Type":"ContainerStarted","Data":"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0"} Oct 31 01:06:24.534071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:24.534074 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" event={"ID":"91a4ffcd-25f8-4f6a-b399-6d50086a7978","Type":"ContainerStarted","Data":"0c31d930d2a2ac61f273ceeaa5a29ae6bb8a7711a456c4516cabd0de37b1397c"} Oct 31 01:06:24.578979 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:24.578933 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" podStartSLOduration=1.578920137 podStartE2EDuration="1.578920137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:24.575933963 +0000 UTC m=+2853.025021782" watchObservedRunningTime="2026-10-31 01:06:24.578920137 +0000 UTC m=+2853.028007776" Oct 31 01:06:24.958382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:24.958282 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:26.052624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.051540 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 01:06:26.649858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.649824 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l"] Oct 31 01:06:26.654847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.654823 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.670872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.670846 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:26.678656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.678618 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l"] Oct 31 01:06:26.683495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.683437 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:06:26.721108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.721075 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.721258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.721194 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qld\" (UniqueName: \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.721371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.721346 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.822597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.822564 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.822742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.822633 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qld\" (UniqueName: \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.822742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.822707 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.823337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.823308 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.825043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.825020 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.846545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.846511 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qld\" (UniqueName: \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\") pod \"maas-api-revoke-sub-bcc36b9e-dtl4l\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:26.958975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.958894 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 01:06:26.959167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.959121 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerName="maas-api" containerID="cri-o://fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c" gracePeriod=30 Oct 31 01:06:26.993774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:26.993716 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:27.048160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.048115 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.179:8443/health\": dial tcp 10.132.0.179:8443: connect: connection refused" Oct 31 01:06:27.361260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.361003 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l"] Oct 31 01:06:27.512189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.512107 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 01:06:27.570872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.570833 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" event={"ID":"492abce6-025c-471b-b86e-f6377f3e011d","Type":"ContainerStarted","Data":"2c2416e8263a5c35bd65069a2f82a5473cd5f78976e967cf12dc0b86147b4f4d"} Oct 31 01:06:27.572566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.572532 5011 generic.go:356] "Generic (PLEG): container finished" podID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerID="fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c" exitCode=0 Oct 31 01:06:27.572833 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.572814 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" event={"ID":"490768d5-5b79-4ed7-8620-bf21e060eb6c","Type":"ContainerDied","Data":"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c"} Oct 31 01:06:27.572912 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.572865 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" event={"ID":"490768d5-5b79-4ed7-8620-bf21e060eb6c","Type":"ContainerDied","Data":"6574d70fb6efb627f3aa9be6d45ac0d2d8a8bda01ff2138e2543f6b31abbd4ef"} Oct 31 01:06:27.572912 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.572894 5011 scope.go:122] "RemoveContainer" containerID="fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c" Oct 31 01:06:27.573075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.573060 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9" Oct 31 01:06:27.590406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.589998 5011 scope.go:122] "RemoveContainer" containerID="fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c" Oct 31 01:06:27.590406 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:06:27.590312 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c\": container with ID starting with fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c not found: ID does not exist" containerID="fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c" Oct 31 01:06:27.590406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.590345 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c"} err="failed to get container status \"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c\": rpc error: code = NotFound desc = could not find container \"fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c\": container with ID starting with fcd630161ddd79ccacbaaa0353951255791a622d824fe0cdf3d729dd8318499c not found: ID does not exist" Oct 31 01:06:27.662651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.662592 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\") pod \"490768d5-5b79-4ed7-8620-bf21e060eb6c\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " Oct 31 01:06:27.662886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.662868 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\") pod \"490768d5-5b79-4ed7-8620-bf21e060eb6c\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " Oct 31 01:06:27.662963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.662902 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\" (UniqueName: \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\") pod \"490768d5-5b79-4ed7-8620-bf21e060eb6c\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " Oct 31 01:06:27.663029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.662968 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") pod \"490768d5-5b79-4ed7-8620-bf21e060eb6c\" (UID: \"490768d5-5b79-4ed7-8620-bf21e060eb6c\") " Oct 31 01:06:27.663286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.663261 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca" pod "490768d5-5b79-4ed7-8620-bf21e060eb6c" (UID: "490768d5-5b79-4ed7-8620-bf21e060eb6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:27.663591 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.663569 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/490768d5-5b79-4ed7-8620-bf21e060eb6c-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:27.665701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.665677 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs" pod "490768d5-5b79-4ed7-8620-bf21e060eb6c" (UID: "490768d5-5b79-4ed7-8620-bf21e060eb6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:27.665872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.665846 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6" pod "490768d5-5b79-4ed7-8620-bf21e060eb6c" (UID: "490768d5-5b79-4ed7-8620-bf21e060eb6c"). InnerVolumeSpecName "kube-api-access-tnlt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:27.665988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.665970 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls" pod "490768d5-5b79-4ed7-8620-bf21e060eb6c" (UID: "490768d5-5b79-4ed7-8620-bf21e060eb6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:27.686815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.686791 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l"] Oct 31 01:06:27.692691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.692663 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82b5e8-9r77l"] Oct 31 01:06:27.765960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.765870 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnlt6\" (UniqueName: \"kubernetes.io/projected/490768d5-5b79-4ed7-8620-bf21e060eb6c-kube-api-access-tnlt6\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:27.765960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.765904 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:27.765960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.765917 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/490768d5-5b79-4ed7-8620-bf21e060eb6c-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:27.915545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.915513 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 01:06:27.919069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:27.919038 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82b5e8-5695f767d5-k8hm9"] Oct 31 01:06:28.582051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:28.582013 5011 generic.go:356] "Generic (PLEG): container finished" podID="492abce6-025c-471b-b86e-f6377f3e011d" containerID="9dcb7dbf4232f67d7a070ef1fe5d1c06571464cddd080966a3814f0b6aa25cf5" exitCode=0 Oct 31 01:06:28.582051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:28.582054 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" event={"ID":"492abce6-025c-471b-b86e-f6377f3e011d","Type":"ContainerDied","Data":"9dcb7dbf4232f67d7a070ef1fe5d1c06571464cddd080966a3814f0b6aa25cf5"} Oct 31 01:06:29.684914 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.684876 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b79504e-b7fb-4fc4-b9e5-c15b64f99610" path="/var/lib/kubelet/pods/2b79504e-b7fb-4fc4-b9e5-c15b64f99610/volumes" Oct 31 01:06:29.685441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.685424 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" path="/var/lib/kubelet/pods/490768d5-5b79-4ed7-8620-bf21e060eb6c/volumes" Oct 31 01:06:29.751144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.751120 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:29.890096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.890012 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\" (UniqueName: \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\") pod \"492abce6-025c-471b-b86e-f6377f3e011d\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " Oct 31 01:06:29.890261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.890123 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\") pod \"492abce6-025c-471b-b86e-f6377f3e011d\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " Oct 31 01:06:29.890261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.890186 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\") pod \"492abce6-025c-471b-b86e-f6377f3e011d\" (UID: \"492abce6-025c-471b-b86e-f6377f3e011d\") " Oct 31 01:06:29.890545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.890512 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca" pod "492abce6-025c-471b-b86e-f6377f3e011d" (UID: "492abce6-025c-471b-b86e-f6377f3e011d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:29.890866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.890847 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:29.892272 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.892246 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca" pod "492abce6-025c-471b-b86e-f6377f3e011d" (UID: "492abce6-025c-471b-b86e-f6377f3e011d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:29.892377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.892357 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld" pod "492abce6-025c-471b-b86e-f6377f3e011d" (UID: "492abce6-025c-471b-b86e-f6377f3e011d"). InnerVolumeSpecName "kube-api-access-68qld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:29.995182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.995142 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qld\" (UniqueName: \"kubernetes.io/projected/492abce6-025c-471b-b86e-f6377f3e011d-kube-api-access-68qld\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:29.995182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:29.995176 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492abce6-025c-471b-b86e-f6377f3e011d-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:30.282680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:30.282604 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:06:30.600516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:30.600488 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" Oct 31 01:06:30.600683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:30.600528 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l" event={"ID":"492abce6-025c-471b-b86e-f6377f3e011d","Type":"ContainerDied","Data":"2c2416e8263a5c35bd65069a2f82a5473cd5f78976e967cf12dc0b86147b4f4d"} Oct 31 01:06:30.600683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:30.600579 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c2416e8263a5c35bd65069a2f82a5473cd5f78976e967cf12dc0b86147b4f4d" Oct 31 01:06:31.948591 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:06:31.948554 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:06:32.664122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664091 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:06:32.664662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664647 5011 cpu_manager.go:414] "containerMap: removing container" podUID="492abce6-025c-471b-b86e-f6377f3e011d" containerName="revoke-keys" Oct 31 01:06:32.664707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664664 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492abce6-025c-471b-b86e-f6377f3e011d" containerName="revoke-keys" Oct 31 01:06:32.664758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664709 5011 cpu_manager.go:414] "containerMap: removing container" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerName="maas-api" Oct 31 01:06:32.664758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664715 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerName="maas-api" Oct 31 01:06:32.664826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664808 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="490768d5-5b79-4ed7-8620-bf21e060eb6c" containerName="maas-api" Oct 31 01:06:32.664826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.664818 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="492abce6-025c-471b-b86e-f6377f3e011d" containerName="revoke-keys" Oct 31 01:06:32.668612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.668589 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.694918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.694891 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-b8fd5a\"/\"kube-root-ca.crt\"" Oct 31 01:06:32.695509 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.695479 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:06:32.695653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.695606 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-b8fd5a\"/\"openshift-service-ca.crt\"" Oct 31 01:06:32.700984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.700950 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-b8fd5a\"/\"auth-test-model-b8fd5a-kserve-self-signed-certs\"" Oct 31 01:06:32.701117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.701036 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-b8fd5a\"/\"default-dockercfg-qlwhh\"" Oct 31 01:06:32.737530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737504 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.737671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737541 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.737738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737725 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.737786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737759 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.737832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737823 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgpq\" (UniqueName: \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.737945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.737923 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839298 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839330 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgpq\" (UniqueName: \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839477 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839594 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839567 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839599 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839725 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.839794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.839752 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.840139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.840110 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.842295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.842269 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.842543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.842519 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.864326 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.864296 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgpq\" (UniqueName: \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\") pod \"auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:32.994483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:32.994379 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:33.327093 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:33.327058 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48733644_b6af_40f5_8810_ddff6a976c8a.slice/crio-3257e580ffbbcad45832fd0c6859b724ead51963e405d97138a958efadf2e973 WatchSource:0}: Error finding container 3257e580ffbbcad45832fd0c6859b724ead51963e405d97138a958efadf2e973: Status 404 returned error can't find the container with id 3257e580ffbbcad45832fd0c6859b724ead51963e405d97138a958efadf2e973 Oct 31 01:06:33.328030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.327998 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:06:33.626611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.626513 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" event={"ID":"48733644-b6af-40f5-8810-ddff6a976c8a","Type":"ContainerStarted","Data":"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff"} Oct 31 01:06:33.626611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.626577 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" event={"ID":"48733644-b6af-40f5-8810-ddff6a976c8a","Type":"ContainerStarted","Data":"3257e580ffbbcad45832fd0c6859b724ead51963e405d97138a958efadf2e973"} Oct 31 01:06:33.680278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.680197 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" podStartSLOduration=1.680184633 podStartE2EDuration="1.680184633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:33.678994062 +0000 UTC m=+2862.128081701" watchObservedRunningTime="2026-10-31 01:06:33.680184633 +0000 UTC m=+2862.129272271" Oct 31 01:06:33.959182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.959100 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:33.977884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.977856 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:33.998232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:33.994950 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:34.649457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.649428 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:06:34.916681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.916583 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:06:34.922584 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.922557 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:34.942697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.942663 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:06:34.943592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.943565 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-99a5c2\"/\"auth-test-model-99a5c2-kserve-self-signed-certs\"" Oct 31 01:06:34.943728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.943649 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-99a5c2\"/\"kube-root-ca.crt\"" Oct 31 01:06:34.943787 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.943725 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-99a5c2\"/\"openshift-service-ca.crt\"" Oct 31 01:06:34.943852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:34.943811 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-99a5c2\"/\"default-dockercfg-t5c5g\"" Oct 31 01:06:35.071510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071462 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.072004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071515 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.072004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071722 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.072004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071793 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvdr\" (UniqueName: \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.072004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071864 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.072004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.071975 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175062 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175158 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvdr\" (UniqueName: \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175247 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175348 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175463 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.175488 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.176074 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.177874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.177846 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.178119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.178099 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.197443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.193044 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.197443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.195102 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.216115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.216081 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvdr\" (UniqueName: \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\") pod \"auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.250300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.249453 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:35.586570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.586526 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:06:35.587766 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:35.587737 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ce1288_cc3e_483e_bdea_8e8eb0036dea.slice/crio-ea9b33584220ec25890514d9ab0d37bdb22a2865822832e963d1e112614ae010 WatchSource:0}: Error finding container ea9b33584220ec25890514d9ab0d37bdb22a2865822832e963d1e112614ae010: Status 404 returned error can't find the container with id ea9b33584220ec25890514d9ab0d37bdb22a2865822832e963d1e112614ae010 Oct 31 01:06:35.660047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:35.660016 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" event={"ID":"05ce1288-cc3e-483e-bdea-8e8eb0036dea","Type":"ContainerStarted","Data":"ea9b33584220ec25890514d9ab0d37bdb22a2865822832e963d1e112614ae010"} Oct 31 01:06:36.673751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:36.673708 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" event={"ID":"05ce1288-cc3e-483e-bdea-8e8eb0036dea","Type":"ContainerStarted","Data":"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6"} Oct 31 01:06:36.728467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:36.728398 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" podStartSLOduration=2.728378187 podStartE2EDuration="2.728378187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:36.723578779 +0000 UTC m=+2865.172666418" watchObservedRunningTime="2026-10-31 01:06:36.728378187 +0000 UTC m=+2865.177465834" Oct 31 01:06:37.250499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.250461 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:37.854433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.854400 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:06:37.859907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.859881 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.893577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.893543 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:06:37.908558 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.908529 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d10721-openshift-default-dockercfg-f7nsh\"" Oct 31 01:06:37.929025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.928994 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929236 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929388 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929496 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fdt\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929719 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929704 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929834 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.929989 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.929973 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.930089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.930076 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:37.930199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:37.930185 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.031661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.031622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.032537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.032491 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.032800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.032777 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.032951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.032935 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.033099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.033083 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.033341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.033304 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.033525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.033501 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.033648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.033633 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.033772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.033752 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fdt\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.034383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.032366 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.040222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.040179 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.041877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.040987 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.042019 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.041227 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.042115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.041818 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.068006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.067938 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fdt\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.069057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.069030 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.084229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.077887 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.088583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.088557 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\") pod \"e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.176328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.176228 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:38.529676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.529640 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:06:38.530116 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:38.530085 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261e6357_4d20_4a20_be52_3e0f630fed32.slice/crio-48873a5646cd7519242ad7bf13c092fb2b4e3345a0050dc0dee367fb2e538c4a WatchSource:0}: Error finding container 48873a5646cd7519242ad7bf13c092fb2b4e3345a0050dc0dee367fb2e538c4a: Status 404 returned error can't find the container with id 48873a5646cd7519242ad7bf13c092fb2b4e3345a0050dc0dee367fb2e538c4a Oct 31 01:06:38.533346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.533305 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:38.533455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.533393 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:38.533455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.533435 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:38.693893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.693850 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerStarted","Data":"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4"} Oct 31 01:06:38.694090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.693915 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerStarted","Data":"48873a5646cd7519242ad7bf13c092fb2b4e3345a0050dc0dee367fb2e538c4a"} Oct 31 01:06:38.747125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:38.747067 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podStartSLOduration=1.747047402 podStartE2EDuration="1.747047402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:38.74532501 +0000 UTC m=+2867.194412650" watchObservedRunningTime="2026-10-31 01:06:38.747047402 +0000 UTC m=+2867.196135038" Oct 31 01:06:39.176806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:39.176761 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:39.185618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:39.185590 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:39.701893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:39.701858 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:39.702999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:39.702963 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:06:41.084963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.084916 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" podUID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" containerName="main" containerID="cri-o://1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007" gracePeriod=45 Oct 31 01:06:41.395439 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.395409 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 01:06:41.501540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501509 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501561 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\" (UniqueName: \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501585 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501675 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\" (UniqueName: \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501935 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501935 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501816 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\") pod \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\" (UID: \"8524ac1c-f5d3-438e-9db7-61a03d95eddb\") " Oct 31 01:06:41.501935 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.501884 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.502103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.502074 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.502254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.502217 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.502624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.502599 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.502737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.502628 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.502737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.502641 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.504271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.504242 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.504363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.504310 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "kube-api-access-mb6cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:41.504413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.504395 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs" pod "8524ac1c-f5d3-438e-9db7-61a03d95eddb" (UID: "8524ac1c-f5d3-438e-9db7-61a03d95eddb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:41.604273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.604192 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8524ac1c-f5d3-438e-9db7-61a03d95eddb-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.604273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.604256 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8524ac1c-f5d3-438e-9db7-61a03d95eddb-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.604482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.604282 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb6cw\" (UniqueName: \"kubernetes.io/projected/8524ac1c-f5d3-438e-9db7-61a03d95eddb-kube-api-access-mb6cw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.722467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.722437 5011 generic.go:356] "Generic (PLEG): container finished" podID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" containerID="1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007" exitCode=0 Oct 31 01:06:41.722607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.722494 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" Oct 31 01:06:41.722607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.722509 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" event={"ID":"8524ac1c-f5d3-438e-9db7-61a03d95eddb","Type":"ContainerDied","Data":"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007"} Oct 31 01:06:41.722607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.722553 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2" event={"ID":"8524ac1c-f5d3-438e-9db7-61a03d95eddb","Type":"ContainerDied","Data":"63145084195b8e0f4c5c0ef27dc76e5b83748463c6cc113d73353cfbcd68d01e"} Oct 31 01:06:41.722607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.722571 5011 scope.go:122] "RemoveContainer" containerID="1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007" Oct 31 01:06:41.734283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.734264 5011 scope.go:122] "RemoveContainer" containerID="1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007" Oct 31 01:06:41.734579 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:06:41.734556 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007\": container with ID starting with 1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007 not found: ID does not exist" containerID="1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007" Oct 31 01:06:41.734670 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.734584 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007"} err="failed to get container status \"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007\": rpc error: code = NotFound desc = could not find container \"1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007\": container with ID starting with 1afafe850c45a542f8cc8356231c9d33a1249fc3baeb461390dc0ec3ec978007 not found: ID does not exist" Oct 31 01:06:41.779373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.779315 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 01:06:41.793132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:41.793102 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-57e594/test-model-57e594-kserve-55bb9477-pkld2"] Oct 31 01:06:42.994992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:42.994961 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:43.010112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:43.010083 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:43.683547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:43.683505 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" path="/var/lib/kubelet/pods/8524ac1c-f5d3-438e-9db7-61a03d95eddb/volumes" Oct 31 01:06:43.751261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:43.751230 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:06:44.433751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.432736 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 01:06:44.703178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.703100 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm"] Oct 31 01:06:44.703947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.703930 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" containerName="main" Oct 31 01:06:44.704010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.703949 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" containerName="main" Oct 31 01:06:44.704106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.704095 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8524ac1c-f5d3-438e-9db7-61a03d95eddb" containerName="main" Oct 31 01:06:44.709214 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.709189 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.723537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.723511 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm"] Oct 31 01:06:44.725589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.725552 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:06:44.726188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.726170 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:44.761696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.761241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.761696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.761336 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.761696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.761377 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxxh\" (UniqueName: \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.863013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.862970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.863189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.863089 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.863189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.863135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxxh\" (UniqueName: \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.863980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.863932 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.865627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.865600 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:44.886624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:44.886599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxxh\" (UniqueName: \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\") pod \"maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:45.024690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.024604 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:45.249889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.249856 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:45.262758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.262721 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:45.339244 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:45.338073 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb2bb1f_3574_4a07_b8a2_ce1ef50e3840.slice/crio-c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f WatchSource:0}: Error finding container c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f: Status 404 returned error can't find the container with id c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f Oct 31 01:06:45.339244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.338414 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm"] Oct 31 01:06:45.787067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.787028 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" event={"ID":"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840","Type":"ContainerStarted","Data":"3a9b5459c57e18551bbc1241befd9ad8f3c031e06f18b75a5bef78966b1c435b"} Oct 31 01:06:45.787510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.787100 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" event={"ID":"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840","Type":"ContainerStarted","Data":"c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f"} Oct 31 01:06:45.799627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.799601 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:06:45.834113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:45.834055 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" podStartSLOduration=1.8340380710000002 podStartE2EDuration="1.834038071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:45.828519907 +0000 UTC m=+2874.277607546" watchObservedRunningTime="2026-10-31 01:06:45.834038071 +0000 UTC m=+2874.283125710" Oct 31 01:06:46.796425 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:46.796344 5011 generic.go:356] "Generic (PLEG): container finished" podID="4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" containerID="3a9b5459c57e18551bbc1241befd9ad8f3c031e06f18b75a5bef78966b1c435b" exitCode=0 Oct 31 01:06:46.796788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:46.796426 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" event={"ID":"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840","Type":"ContainerDied","Data":"3a9b5459c57e18551bbc1241befd9ad8f3c031e06f18b75a5bef78966b1c435b"} Oct 31 01:06:47.962425 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:47.962391 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:48.102286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.102249 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\" (UniqueName: \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\") pod \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " Oct 31 01:06:48.102482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.102362 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\") pod \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " Oct 31 01:06:48.102554 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.102505 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\") pod \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\" (UID: \"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840\") " Oct 31 01:06:48.103383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.103343 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca" pod "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" (UID: "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:48.105038 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.105009 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca" pod "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" (UID: "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:48.107427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.107399 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh" pod "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" (UID: "4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840"). InnerVolumeSpecName "kube-api-access-ngxxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:48.203738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.203707 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngxxh\" (UniqueName: \"kubernetes.io/projected/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-kube-api-access-ngxxh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:48.203738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.203734 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:48.203738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.203744 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:48.813190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.813156 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" event={"ID":"4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840","Type":"ContainerDied","Data":"c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f"} Oct 31 01:06:48.813190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.813179 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm" Oct 31 01:06:48.813409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:48.813218 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5517d826d3199fc7d084d4e56b382ac50bac2f06e37144cc47a3cc3a00d4c0f" Oct 31 01:06:50.173220 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.173175 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:06:50.173764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.173749 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" containerName="revoke-keys" Oct 31 01:06:50.173818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.173764 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" containerName="revoke-keys" Oct 31 01:06:50.173878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.173869 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" containerName="revoke-keys" Oct 31 01:06:50.177596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.177570 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.192535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.192511 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-498580-openshift-default-dockercfg-8p4s8\"" Oct 31 01:06:50.208977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.208945 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:06:50.342951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.342915 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9dk\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.342954 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343282 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343334 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343450 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343503 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343563 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.343636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.343590 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444536 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9dk\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444569 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444604 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444667 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444693 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444883 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444939 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.444973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444965 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.444990 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.445138 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.445300 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.445395 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.445490 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.445792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.445766 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.447360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.447334 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.447730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.447713 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.468481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.468449 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.468791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.468772 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9dk\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.492088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.492064 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:50.827349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.819946 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:06:50.845260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.843591 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:50.845260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.843718 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:50.845260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:50.843761 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:06:51.864063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:51.864023 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" event={"ID":"e74c9431-541e-4105-9950-20b2b24b50f8","Type":"ContainerStarted","Data":"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed"} Oct 31 01:06:51.865370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:51.864641 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" event={"ID":"e74c9431-541e-4105-9950-20b2b24b50f8","Type":"ContainerStarted","Data":"5058002db3221e32c4e63c1c146efa52d468671d18c778b1ef903c13b69b50b3"} Oct 31 01:06:51.924685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:51.923927 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podStartSLOduration=1.923907799 podStartE2EDuration="1.923907799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:51.922387422 +0000 UTC m=+2880.371475102" watchObservedRunningTime="2026-10-31 01:06:51.923907799 +0000 UTC m=+2880.372995439" Oct 31 01:06:52.492861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:52.492825 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:52.494257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:52.494226 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:52.494398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:52.494287 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:53.493358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:53.493320 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:53.493774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:53.493379 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:53.812252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:53.812161 5011 scope.go:122] "RemoveContainer" containerID="7a7aca03d153343167e75616f6fbb1dc429a860bcd518af78916be0b2af18e7b" Oct 31 01:06:54.493121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:54.493061 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:54.493324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:54.493142 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:55.493274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:55.493196 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:55.493689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:55.493290 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:56.492944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.492904 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:56.493143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.492963 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:56.793620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.793539 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7"] Oct 31 01:06:56.799576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.799555 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.814391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.814366 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:56.815956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.815931 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7"] Oct 31 01:06:56.844265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.832833 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:06:56.878424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.878390 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.878744 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.878717 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.878907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.878888 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xsr\" (UniqueName: \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.980177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.980143 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.980370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.980201 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.980370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.980253 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xsr\" (UniqueName: \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.980875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.980835 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:56.983050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:56.983026 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:57.003797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.003773 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xsr\" (UniqueName: \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\") pod \"maas-api-revoke-sub-aa2181a5-w6bx7\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:57.116992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.114726 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:06:57.466173 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:06:57.466120 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1696aea6_23c9_46ed_92f9_0bcfa4b08284.slice/crio-fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000 WatchSource:0}: Error finding container fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000: Status 404 returned error can't find the container with id fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000 Oct 31 01:06:57.466337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.466229 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7"] Oct 31 01:06:57.493368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.493340 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:06:57.493465 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.493394 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 01:06:57.975797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.975761 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" event={"ID":"1696aea6-23c9-46ed-92f9-0bcfa4b08284","Type":"ContainerStarted","Data":"4f10c263a0c4b4527df8a5b5b753d113c4ec615876815f4c9b953cee9966341a"} Oct 31 01:06:57.976317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:57.975818 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" event={"ID":"1696aea6-23c9-46ed-92f9-0bcfa4b08284","Type":"ContainerStarted","Data":"fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000"} Oct 31 01:06:58.019513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.019457 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" podStartSLOduration=2.019440065 podStartE2EDuration="2.019440065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:58.016615709 +0000 UTC m=+2886.465703346" watchObservedRunningTime="2026-10-31 01:06:58.019440065 +0000 UTC m=+2886.468527706" Oct 31 01:06:58.497652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.497625 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:58.498123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.498097 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:58.498764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.498741 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:06:58.985867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.985828 5011 generic.go:356] "Generic (PLEG): container finished" podID="1696aea6-23c9-46ed-92f9-0bcfa4b08284" containerID="4f10c263a0c4b4527df8a5b5b753d113c4ec615876815f4c9b953cee9966341a" exitCode=0 Oct 31 01:06:58.986349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:58.985916 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" event={"ID":"1696aea6-23c9-46ed-92f9-0bcfa4b08284","Type":"ContainerDied","Data":"4f10c263a0c4b4527df8a5b5b753d113c4ec615876815f4c9b953cee9966341a"} Oct 31 01:06:59.468467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.468395 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" podUID="f6ebc792-aff3-4ac8-8872-5936bf23292d" containerName="main" containerID="cri-o://bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18" gracePeriod=45 Oct 31 01:06:59.773326 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.773297 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:06:59.817076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817043 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817161 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817241 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817270 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817322 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:59.817401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817331 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817403 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:59.817647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817574 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\" (UniqueName: \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\") pod \"f6ebc792-aff3-4ac8-8872-5936bf23292d\" (UID: \"f6ebc792-aff3-4ac8-8872-5936bf23292d\") " Oct 31 01:06:59.817647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.817580 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:59.818256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.818230 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.818450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.818425 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.818575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.818480 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.819888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.819862 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "kube-api-access-v2n5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:59.819983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.819868 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:59.824072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.824042 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm" pod "f6ebc792-aff3-4ac8-8872-5936bf23292d" (UID: "f6ebc792-aff3-4ac8-8872-5936bf23292d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:59.920262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.920229 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ebc792-aff3-4ac8-8872-5936bf23292d-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.920262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.920260 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ebc792-aff3-4ac8-8872-5936bf23292d-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.920441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.920275 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n5l\" (UniqueName: \"kubernetes.io/projected/f6ebc792-aff3-4ac8-8872-5936bf23292d-kube-api-access-v2n5l\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:06:59.994839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.994759 5011 generic.go:356] "Generic (PLEG): container finished" podID="f6ebc792-aff3-4ac8-8872-5936bf23292d" containerID="bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18" exitCode=0 Oct 31 01:06:59.994839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.994794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" event={"ID":"f6ebc792-aff3-4ac8-8872-5936bf23292d","Type":"ContainerDied","Data":"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18"} Oct 31 01:06:59.994839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.994822 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" Oct 31 01:06:59.995405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.994848 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs" event={"ID":"f6ebc792-aff3-4ac8-8872-5936bf23292d","Type":"ContainerDied","Data":"85625da99e60b3ab73899e8f479242a16522d663029e4b1acd203513738ac30d"} Oct 31 01:06:59.995405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:06:59.994872 5011 scope.go:122] "RemoveContainer" containerID="bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18" Oct 31 01:07:00.006432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.006409 5011 scope.go:122] "RemoveContainer" containerID="bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18" Oct 31 01:07:00.006713 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:07:00.006695 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18\": container with ID starting with bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18 not found: ID does not exist" containerID="bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18" Oct 31 01:07:00.006770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.006721 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18"} err="failed to get container status \"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18\": rpc error: code = NotFound desc = could not find container \"bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18\": container with ID starting with bfba2b0bf87da42c9d046a7df5c93927845234ef1ddd5e772067b14799f6bf18 not found: ID does not exist" Oct 31 01:07:00.051377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.051341 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 01:07:00.057274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.057243 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4f4e69/test-model-4f4e69-kserve-6d655d7775-wthqs"] Oct 31 01:07:00.152845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.152816 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:07:00.227029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.226983 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\") pod \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " Oct 31 01:07:00.227250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.227113 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\" (UniqueName: \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\") pod \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " Oct 31 01:07:00.227685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.227333 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\") pod \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\" (UID: \"1696aea6-23c9-46ed-92f9-0bcfa4b08284\") " Oct 31 01:07:00.228360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.228337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca" pod "1696aea6-23c9-46ed-92f9-0bcfa4b08284" (UID: "1696aea6-23c9-46ed-92f9-0bcfa4b08284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:00.232699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.230834 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca" pod "1696aea6-23c9-46ed-92f9-0bcfa4b08284" (UID: "1696aea6-23c9-46ed-92f9-0bcfa4b08284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:00.232899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.232875 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr" pod "1696aea6-23c9-46ed-92f9-0bcfa4b08284" (UID: "1696aea6-23c9-46ed-92f9-0bcfa4b08284"). InnerVolumeSpecName "kube-api-access-s2xsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:00.328815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.328789 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:00.328815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.328815 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1696aea6-23c9-46ed-92f9-0bcfa4b08284-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:00.328815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:00.328828 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xsr\" (UniqueName: \"kubernetes.io/projected/1696aea6-23c9-46ed-92f9-0bcfa4b08284-kube-api-access-s2xsr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:01.004999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:01.004959 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" event={"ID":"1696aea6-23c9-46ed-92f9-0bcfa4b08284","Type":"ContainerDied","Data":"fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000"} Oct 31 01:07:01.005494 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:01.005028 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3ec0b773f72e57bca99dbd1e0e8386ded00e2733b742edd2c98276e63ad000" Oct 31 01:07:01.005494 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:01.004976 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7" Oct 31 01:07:01.683498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:01.683465 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ebc792-aff3-4ac8-8872-5936bf23292d" path="/var/lib/kubelet/pods/f6ebc792-aff3-4ac8-8872-5936bf23292d/volumes" Oct 31 01:07:01.948987 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:07:01.948895 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:07:13.534117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:13.534073 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 01:07:13.534973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:13.534929 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" podUID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" containerName="maas-api" containerID="cri-o://533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb" gracePeriod=30 Oct 31 01:07:14.115501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.114775 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 01:07:14.133459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.130162 5011 generic.go:356] "Generic (PLEG): container finished" podID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" containerID="533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb" exitCode=0 Oct 31 01:07:14.133459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.130294 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" event={"ID":"8654b8ab-b4d0-4f9a-8481-01f663f5c234","Type":"ContainerDied","Data":"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb"} Oct 31 01:07:14.133459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.130344 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" event={"ID":"8654b8ab-b4d0-4f9a-8481-01f663f5c234","Type":"ContainerDied","Data":"19b0f924085be17fedd528957826e2b2cc30e9f23a933fead46e8c5d09ba7cb5"} Oct 31 01:07:14.133459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.130374 5011 scope.go:122] "RemoveContainer" containerID="533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb" Oct 31 01:07:14.133459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.130527 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2" Oct 31 01:07:14.145448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.145405 5011 scope.go:122] "RemoveContainer" containerID="533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb" Oct 31 01:07:14.145792 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:07:14.145757 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb\": container with ID starting with 533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb not found: ID does not exist" containerID="533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb" Oct 31 01:07:14.145938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.145802 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb"} err="failed to get container status \"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb\": rpc error: code = NotFound desc = could not find container \"533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb\": container with ID starting with 533ce6c80db20b379c819b0dc261f2e8cbe307594e597a96acbb41fc7370caeb not found: ID does not exist" Oct 31 01:07:14.215564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.215532 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\") pod \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " Oct 31 01:07:14.215727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.215651 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\") pod \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " Oct 31 01:07:14.215780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.215764 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\" (UniqueName: \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\") pod \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " Oct 31 01:07:14.215854 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.215838 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\") pod \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\" (UID: \"8654b8ab-b4d0-4f9a-8481-01f663f5c234\") " Oct 31 01:07:14.216068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.216042 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca" pod "8654b8ab-b4d0-4f9a-8481-01f663f5c234" (UID: "8654b8ab-b4d0-4f9a-8481-01f663f5c234"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:14.216338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.216318 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8654b8ab-b4d0-4f9a-8481-01f663f5c234-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:14.218131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.217701 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs" pod "8654b8ab-b4d0-4f9a-8481-01f663f5c234" (UID: "8654b8ab-b4d0-4f9a-8481-01f663f5c234"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:14.218131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.218029 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv" pod "8654b8ab-b4d0-4f9a-8481-01f663f5c234" (UID: "8654b8ab-b4d0-4f9a-8481-01f663f5c234"). InnerVolumeSpecName "kube-api-access-fsbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:14.218131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.218029 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls" pod "8654b8ab-b4d0-4f9a-8481-01f663f5c234" (UID: "8654b8ab-b4d0-4f9a-8481-01f663f5c234"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:14.223749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.223721 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm"] Oct 31 01:07:14.227955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.227764 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-57e594-zcjbm"] Oct 31 01:07:14.318517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.318484 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbjv\" (UniqueName: \"kubernetes.io/projected/8654b8ab-b4d0-4f9a-8481-01f663f5c234-kube-api-access-fsbjv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:14.318517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.318516 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:14.318798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.318527 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8654b8ab-b4d0-4f9a-8481-01f663f5c234-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:14.477671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.477642 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 01:07:14.489621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:14.489589 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-57e594-bfb8bbbc9-z94j2"] Oct 31 01:07:15.683855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.683820 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840" path="/var/lib/kubelet/pods/4eb2bb1f-3574-4a07-b8a2-ce1ef50e3840/volumes" Oct 31 01:07:15.684377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.684357 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" path="/var/lib/kubelet/pods/8654b8ab-b4d0-4f9a-8481-01f663f5c234/volumes" Oct 31 01:07:15.883489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.883455 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:07:15.884012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.883999 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f6ebc792-aff3-4ac8-8872-5936bf23292d" containerName="main" Oct 31 01:07:15.884058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884013 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ebc792-aff3-4ac8-8872-5936bf23292d" containerName="main" Oct 31 01:07:15.884058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884035 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1696aea6-23c9-46ed-92f9-0bcfa4b08284" containerName="revoke-keys" Oct 31 01:07:15.884058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884041 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1696aea6-23c9-46ed-92f9-0bcfa4b08284" containerName="revoke-keys" Oct 31 01:07:15.884058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884048 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" containerName="maas-api" Oct 31 01:07:15.884058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884053 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" containerName="maas-api" Oct 31 01:07:15.884293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884145 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ebc792-aff3-4ac8-8872-5936bf23292d" containerName="main" Oct 31 01:07:15.884293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884155 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1696aea6-23c9-46ed-92f9-0bcfa4b08284" containerName="revoke-keys" Oct 31 01:07:15.884293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.884161 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8654b8ab-b4d0-4f9a-8481-01f663f5c234" containerName="maas-api" Oct 31 01:07:15.889254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.889229 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:15.913169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.912885 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d10721-serving-cert\"" Oct 31 01:07:15.913169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.912998 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:07:15.940349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.940018 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7c5q\" (UniqueName: \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:15.940349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.940114 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:15.940349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.940180 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:15.940349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:15.940254 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.041474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.041439 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.041668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.041496 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.041668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.041647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7c5q\" (UniqueName: \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.041785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.041702 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.042259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.042200 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.044176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.044153 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.044327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.044303 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.069937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.069908 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7c5q\" (UniqueName: \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\") pod \"maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.207746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.207668 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:16.529562 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:07:16.529528 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068a4a68_e133_4cc8_9a19_dcb0ad036218.slice/crio-ca88484a291489729048eeea2919394a24804d48905574f160c2840eb30e32e5 WatchSource:0}: Error finding container ca88484a291489729048eeea2919394a24804d48905574f160c2840eb30e32e5: Status 404 returned error can't find the container with id ca88484a291489729048eeea2919394a24804d48905574f160c2840eb30e32e5 Oct 31 01:07:16.530217 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.530136 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:07:16.531218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:16.531187 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:07:17.162603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:17.161797 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" event={"ID":"068a4a68-e133-4cc8-9a19-dcb0ad036218","Type":"ContainerStarted","Data":"0d09a58ec65132ffd49e38f78375cc9fbffa15f1ccd34ba76a26fb024bed0ad3"} Oct 31 01:07:17.162603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:17.161893 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" event={"ID":"068a4a68-e133-4cc8-9a19-dcb0ad036218","Type":"ContainerStarted","Data":"ca88484a291489729048eeea2919394a24804d48905574f160c2840eb30e32e5"} Oct 31 01:07:17.162603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:17.162108 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:17.221433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:17.221378 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" podStartSLOduration=1.7029037009999999 podStartE2EDuration="2.221363393s" totalImagesPullingTime="518.459692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:17.218902304 +0000 UTC m=+2905.667989958" watchObservedRunningTime="2026-10-31 01:07:17.221363393 +0000 UTC m=+2905.670451032" Oct 31 01:07:21.489075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.489040 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm"] Oct 31 01:07:21.494078 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.494055 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.510990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.510963 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm"] Oct 31 01:07:21.518787 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.518759 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:21.518965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.518945 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:07:21.656515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.656484 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.656704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.656602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.656704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.656662 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phznv\" (UniqueName: \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.758058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.757980 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.758232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.758069 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.758232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.758109 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phznv\" (UniqueName: \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.758872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.758838 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.760551 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.760524 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.781360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.781335 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phznv\" (UniqueName: \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\") pod \"maas-api-revoke-sub-05222b04-tc6gm\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:21.808881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:21.808857 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:22.168224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.164668 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm"] Oct 31 01:07:22.205634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.205602 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" event={"ID":"df529f99-93cc-425e-8f57-19669e2ddee5","Type":"ContainerStarted","Data":"06cb35ad1e3be1781ddc5da5d2935f72c861cc702151d89875335d502022637e"} Oct 31 01:07:22.496129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.496099 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c"] Oct 31 01:07:22.500178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.500161 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.533925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.529592 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c"] Oct 31 01:07:22.573903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.573873 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlz4\" (UniqueName: \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.574090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.574066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.574136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.574122 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.676089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.676053 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlz4\" (UniqueName: \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.676268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.676235 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.676347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.676292 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.676936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.676916 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.678641 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.678581 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.699065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.699041 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlz4\" (UniqueName: \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\") pod \"maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:22.820159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:22.820129 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:23.149441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:23.149409 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c"] Oct 31 01:07:23.178318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:23.178292 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:07:23.227071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:23.227035 5011 generic.go:356] "Generic (PLEG): container finished" podID="df529f99-93cc-425e-8f57-19669e2ddee5" containerID="489b02a268aedce52785e933da103401ffecbfcd9e2b904c31fe7f93a705a623" exitCode=0 Oct 31 01:07:23.227490 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:23.227443 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" event={"ID":"df529f99-93cc-425e-8f57-19669e2ddee5","Type":"ContainerDied","Data":"489b02a268aedce52785e933da103401ffecbfcd9e2b904c31fe7f93a705a623"} Oct 31 01:07:23.229890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:23.229701 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" event={"ID":"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705","Type":"ContainerStarted","Data":"b81c083aa1bf9ecf14928d13e6359e36bab8aacac9dc9947acfa7c0c485284cb"} Oct 31 01:07:24.241892 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.241855 5011 generic.go:356] "Generic (PLEG): container finished" podID="b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" containerID="4fa8e827ce3c64f2fa627db0b73eb908efc7bf60d72096af9cb73400bde2e81f" exitCode=0 Oct 31 01:07:24.242267 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.241935 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" event={"ID":"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705","Type":"ContainerDied","Data":"4fa8e827ce3c64f2fa627db0b73eb908efc7bf60d72096af9cb73400bde2e81f"} Oct 31 01:07:24.426572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.426536 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:24.460604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.460573 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:07:24.461201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.461180 5011 cpu_manager.go:414] "containerMap: removing container" podUID="df529f99-93cc-425e-8f57-19669e2ddee5" containerName="revoke-keys" Oct 31 01:07:24.461201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.461201 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df529f99-93cc-425e-8f57-19669e2ddee5" containerName="revoke-keys" Oct 31 01:07:24.461447 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.461432 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="df529f99-93cc-425e-8f57-19669e2ddee5" containerName="revoke-keys" Oct 31 01:07:24.464949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.464926 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.486247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.483112 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-328d61\"/\"kube-root-ca.crt\"" Oct 31 01:07:24.487859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.486546 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-328d61\"/\"default-dockercfg-pfh6r\"" Oct 31 01:07:24.487859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.486660 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-328d61\"/\"auth-test-model-328d61-kserve-self-signed-certs\"" Oct 31 01:07:24.487859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.486737 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-328d61\"/\"openshift-service-ca.crt\"" Oct 31 01:07:24.491857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.491831 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:07:24.502146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.502120 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\") pod \"df529f99-93cc-425e-8f57-19669e2ddee5\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " Oct 31 01:07:24.502275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.502183 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\" (UniqueName: \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\") pod \"df529f99-93cc-425e-8f57-19669e2ddee5\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " Oct 31 01:07:24.502322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.502290 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\") pod \"df529f99-93cc-425e-8f57-19669e2ddee5\" (UID: \"df529f99-93cc-425e-8f57-19669e2ddee5\") " Oct 31 01:07:24.502585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.502553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca" pod "df529f99-93cc-425e-8f57-19669e2ddee5" (UID: "df529f99-93cc-425e-8f57-19669e2ddee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:24.502811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.502792 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:24.504372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.504349 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca" pod "df529f99-93cc-425e-8f57-19669e2ddee5" (UID: "df529f99-93cc-425e-8f57-19669e2ddee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:24.504658 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.504639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv" pod "df529f99-93cc-425e-8f57-19669e2ddee5" (UID: "df529f99-93cc-425e-8f57-19669e2ddee5"). InnerVolumeSpecName "kube-api-access-phznv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:24.603816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.603783 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.603816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.603822 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5p7s\" (UniqueName: \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.604046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.603856 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.604046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.604014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.604176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.604152 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.604282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.604259 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.604561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.604541 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phznv\" (UniqueName: \"kubernetes.io/projected/df529f99-93cc-425e-8f57-19669e2ddee5-kube-api-access-phznv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:24.604597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.604569 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df529f99-93cc-425e-8f57-19669e2ddee5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709529 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709754 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709785 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5p7s\" (UniqueName: \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709825 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709879 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.709955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.710486 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.712280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.710742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.721232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.716620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.721232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.719511 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.752232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.737931 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.752232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.744873 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5p7s\" (UniqueName: \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\") pod \"auth-test-model-328d61-kserve-79d9c7495c-2m9jw\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:24.804232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:24.803894 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:25.170926 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:07:25.170891 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf5a9000_b16f_4cf0_8571_ecad8628a7c4.slice/crio-cbe792ab0400de9f291b4ef4f6501f1ad554e0d9e6d3cac0818819218a842d67 WatchSource:0}: Error finding container cbe792ab0400de9f291b4ef4f6501f1ad554e0d9e6d3cac0818819218a842d67: Status 404 returned error can't find the container with id cbe792ab0400de9f291b4ef4f6501f1ad554e0d9e6d3cac0818819218a842d67 Oct 31 01:07:25.173795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.173717 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:07:25.255227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.254203 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" event={"ID":"df5a9000-b16f-4cf0-8571-ecad8628a7c4","Type":"ContainerStarted","Data":"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752"} Oct 31 01:07:25.255227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.254298 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" event={"ID":"df5a9000-b16f-4cf0-8571-ecad8628a7c4","Type":"ContainerStarted","Data":"cbe792ab0400de9f291b4ef4f6501f1ad554e0d9e6d3cac0818819218a842d67"} Oct 31 01:07:25.256473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.256438 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" Oct 31 01:07:25.256585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.256480 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm" event={"ID":"df529f99-93cc-425e-8f57-19669e2ddee5","Type":"ContainerDied","Data":"06cb35ad1e3be1781ddc5da5d2935f72c861cc702151d89875335d502022637e"} Oct 31 01:07:25.256585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.256522 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06cb35ad1e3be1781ddc5da5d2935f72c861cc702151d89875335d502022637e" Oct 31 01:07:25.297796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.297535 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" podStartSLOduration=1.297520348 podStartE2EDuration="1.297520348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:25.296077971 +0000 UTC m=+2913.745165610" watchObservedRunningTime="2026-10-31 01:07:25.297520348 +0000 UTC m=+2913.746607992" Oct 31 01:07:25.445315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.445239 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:25.636074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.636039 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\" (UniqueName: \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\") pod \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " Oct 31 01:07:25.636289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.636101 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\") pod \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " Oct 31 01:07:25.636409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.636387 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\") pod \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\" (UID: \"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705\") " Oct 31 01:07:25.636822 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.636792 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca" pod "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" (UID: "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:25.637061 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.637024 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:25.638432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.638410 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca" pod "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" (UID: "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:25.638631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.638609 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4" pod "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" (UID: "b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705"). InnerVolumeSpecName "kube-api-access-gzlz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:25.738392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.738323 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzlz4\" (UniqueName: \"kubernetes.io/projected/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-kube-api-access-gzlz4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:25.738392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.738350 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:25.804749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:25.804720 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:26.268018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:26.267975 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" event={"ID":"b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705","Type":"ContainerDied","Data":"b81c083aa1bf9ecf14928d13e6359e36bab8aacac9dc9947acfa7c0c485284cb"} Oct 31 01:07:26.268018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:26.268000 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c" Oct 31 01:07:26.268472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:26.268042 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81c083aa1bf9ecf14928d13e6359e36bab8aacac9dc9947acfa7c0c485284cb" Oct 31 01:07:31.949168 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:07:31.949123 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:07:32.970714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:32.970629 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 01:07:34.804576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:34.804544 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:34.820522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:34.820493 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:35.356600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:35.356568 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:07:37.971980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:37.971887 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" podUID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" containerName="payload-processing" containerID="cri-o://35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401" gracePeriod=25 Oct 31 01:07:38.261943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.261911 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 01:07:38.370666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.370627 5011 generic.go:356] "Generic (PLEG): container finished" podID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" containerID="35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401" exitCode=0 Oct 31 01:07:38.370818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.370675 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" Oct 31 01:07:38.370818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.370717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" event={"ID":"ee2f39f9-3b01-4649-9623-0e90d4d8dadb","Type":"ContainerDied","Data":"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401"} Oct 31 01:07:38.370818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.370779 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn" event={"ID":"ee2f39f9-3b01-4649-9623-0e90d4d8dadb","Type":"ContainerDied","Data":"3cd5557cae73b7937997151bfb0f00d09160b1c7d0a180dfe983a5d334ec6762"} Oct 31 01:07:38.370818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.370802 5011 scope.go:122] "RemoveContainer" containerID="35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401" Oct 31 01:07:38.381401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.381380 5011 scope.go:122] "RemoveContainer" containerID="35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401" Oct 31 01:07:38.381673 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:07:38.381653 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401\": container with ID starting with 35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401 not found: ID does not exist" containerID="35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401" Oct 31 01:07:38.381749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.381684 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401"} err="failed to get container status \"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401\": rpc error: code = NotFound desc = could not find container \"35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401\": container with ID starting with 35756465438baa2f576cc0d8babc64799c111fe11f997c462ff77ad136a02401 not found: ID does not exist" Oct 31 01:07:38.410874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.410844 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\") pod \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " Oct 31 01:07:38.411014 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.410981 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\" (UniqueName: \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\") pod \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\" (UID: \"ee2f39f9-3b01-4649-9623-0e90d4d8dadb\") " Oct 31 01:07:38.411200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.411177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume" pod "ee2f39f9-3b01-4649-9623-0e90d4d8dadb" (UID: "ee2f39f9-3b01-4649-9623-0e90d4d8dadb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:38.411514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.411492 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:38.412963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.412944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2" pod "ee2f39f9-3b01-4649-9623-0e90d4d8dadb" (UID: "ee2f39f9-3b01-4649-9623-0e90d4d8dadb"). InnerVolumeSpecName "kube-api-access-8m2q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:38.513364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.513292 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m2q2\" (UniqueName: \"kubernetes.io/projected/ee2f39f9-3b01-4649-9623-0e90d4d8dadb-kube-api-access-8m2q2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:38.723263 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.723227 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 01:07:38.731603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:38.731576 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f4e69-5f8fb86855-mzzfn"] Oct 31 01:07:39.683342 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:39.683306 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" path="/var/lib/kubelet/pods/ee2f39f9-3b01-4649-9623-0e90d4d8dadb/volumes" Oct 31 01:07:52.049191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.049151 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg"] Oct 31 01:07:52.054355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.054234 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25947889-zpjlg"] Oct 31 01:07:52.145246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.145188 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49"] Oct 31 01:07:52.146097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146078 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" containerName="payload-processing" Oct 31 01:07:52.146147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146101 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" containerName="payload-processing" Oct 31 01:07:52.146147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146116 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" containerName="revoke-keys" Oct 31 01:07:52.146147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146125 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" containerName="revoke-keys" Oct 31 01:07:52.146298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146285 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2f39f9-3b01-4649-9623-0e90d4d8dadb" containerName="payload-processing" Oct 31 01:07:52.146340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.146300 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" containerName="revoke-keys" Oct 31 01:07:52.150457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.150434 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.171234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.171148 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:07:52.172991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.172133 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:52.183530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.183499 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49"] Oct 31 01:07:52.196946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.196897 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.197135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.197094 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.197274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.197161 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scczt\" (UniqueName: \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.298391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.298354 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.298580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.298419 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.298580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.298444 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scczt\" (UniqueName: \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.299142 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.299112 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.300841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.300789 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.322573 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.322536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scczt\" (UniqueName: \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\") pod \"maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.488554 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.488526 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:52.797691 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:07:52.797661 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8ed87_9dc1_48fc_ac60_c15c35a818c5.slice/crio-534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8 WatchSource:0}: Error finding container 534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8: Status 404 returned error can't find the container with id 534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8 Oct 31 01:07:52.797842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:52.797779 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49"] Oct 31 01:07:53.512502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:53.512416 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" event={"ID":"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5","Type":"ContainerStarted","Data":"41f10a0414251cc6d96812a60040e7dde68b0ee6d24bf347583a5fccd28a5771"} Oct 31 01:07:53.512861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:53.512519 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" event={"ID":"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5","Type":"ContainerStarted","Data":"534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8"} Oct 31 01:07:53.557248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:53.557165 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" podStartSLOduration=1.557148392 podStartE2EDuration="1.557148392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:53.55440258 +0000 UTC m=+2942.003490219" watchObservedRunningTime="2026-10-31 01:07:53.557148392 +0000 UTC m=+2942.006236032" Oct 31 01:07:53.684049 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:53.684014 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14467936-4c55-4cf2-8ac6-8ba82dff4510" path="/var/lib/kubelet/pods/14467936-4c55-4cf2-8ac6-8ba82dff4510/volumes" Oct 31 01:07:54.521510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:54.521474 5011 generic.go:356] "Generic (PLEG): container finished" podID="c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" containerID="41f10a0414251cc6d96812a60040e7dde68b0ee6d24bf347583a5fccd28a5771" exitCode=0 Oct 31 01:07:54.521946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:54.521559 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" event={"ID":"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5","Type":"ContainerDied","Data":"41f10a0414251cc6d96812a60040e7dde68b0ee6d24bf347583a5fccd28a5771"} Oct 31 01:07:55.710350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.710324 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:55.740116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.740087 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\") pod \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " Oct 31 01:07:55.740321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.740225 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\" (UniqueName: \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\") pod \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " Oct 31 01:07:55.740321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.740282 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\") pod \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\" (UID: \"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5\") " Oct 31 01:07:55.740446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.740425 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca" pod "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" (UID: "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:55.740903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.740880 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:55.742682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.742646 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt" pod "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" (UID: "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5"). InnerVolumeSpecName "kube-api-access-scczt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:55.743288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.743108 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca" pod "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" (UID: "c0d8ed87-9dc1-48fc-ac60-c15c35a818c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:55.842134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.842095 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scczt\" (UniqueName: \"kubernetes.io/projected/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-kube-api-access-scczt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:55.842134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:55.842138 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:07:56.548658 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:56.548614 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" event={"ID":"c0d8ed87-9dc1-48fc-ac60-c15c35a818c5","Type":"ContainerDied","Data":"534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8"} Oct 31 01:07:56.548847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:56.548722 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534ecb2578f9825ef19ecdfb997563302fd55e0370ee94bd12f2dd077ad650f8" Oct 31 01:07:56.548847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:56.548825 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49" Oct 31 01:07:57.197228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:57.192602 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:07:58.909518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.909482 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:07:58.910323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.910297 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" containerName="revoke-keys" Oct 31 01:07:58.910323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.910321 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" containerName="revoke-keys" Oct 31 01:07:58.910550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.910538 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" containerName="revoke-keys" Oct 31 01:07:58.915105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.915081 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:58.944983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.944956 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:07:58.948551 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:58.948524 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ada0c1-openshift-default-dockercfg-7hrph\"" Oct 31 01:07:59.034132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034078 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034149 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034230 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034328 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034403 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034527 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034554 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.034818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.034608 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrwd8\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.146575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.146529 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.146845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.146821 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.146961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.146948 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147062 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrwd8\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147240 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147375 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147499 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147666 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147818 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.147975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.147948 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.155303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.148830 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.155486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.149065 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.155603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.154669 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.155715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.154905 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.189329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.181045 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.189329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.186016 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrwd8\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.189329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.188449 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.192188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.192139 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.239226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.237555 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:07:59.867239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.859481 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c"] Oct 31 01:07:59.871474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.871442 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f4e69-t589c"] Oct 31 01:07:59.903031 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:07:59.903005 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74407c2a_2b5d_4185_955e_b47c4e73d766.slice/crio-ced8e60aacd8f723d4df8d118517ea90e8c661ed922b7e1707a56939ad15c03a WatchSource:0}: Error finding container ced8e60aacd8f723d4df8d118517ea90e8c661ed922b7e1707a56939ad15c03a: Status 404 returned error can't find the container with id ced8e60aacd8f723d4df8d118517ea90e8c661ed922b7e1707a56939ad15c03a Oct 31 01:07:59.904371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.904346 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:07:59.906775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.906738 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:59.906859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.906820 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:59.906899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:07:59.906861 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:00.635230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:00.631168 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerStarted","Data":"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a"} Oct 31 01:08:00.635230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:00.631302 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerStarted","Data":"ced8e60aacd8f723d4df8d118517ea90e8c661ed922b7e1707a56939ad15c03a"} Oct 31 01:08:00.703502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:00.703450 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podStartSLOduration=2.703432972 podStartE2EDuration="2.703432972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:00.689902523 +0000 UTC m=+2949.138990164" watchObservedRunningTime="2026-10-31 01:08:00.703432972 +0000 UTC m=+2949.152520610" Oct 31 01:08:01.241298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:01.241261 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:08:01.250629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:01.250601 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:08:01.654061 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:01.654022 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:08:01.655153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:01.655129 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:08:01.684475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:01.684445 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705" path="/var/lib/kubelet/pods/b0b57b7b-f2eb-4b9f-a21c-1fad91a9a705/volumes" Oct 31 01:08:01.989323 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:08:01.989232 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:08:03.060233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:03.056251 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7"] Oct 31 01:08:03.065225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:03.061864 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7e2e6e8-tsdq7"] Oct 31 01:08:03.685066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:03.685031 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eaa2dc0-5aa3-4b4c-83de-f1618d232941" path="/var/lib/kubelet/pods/2eaa2dc0-5aa3-4b4c-83de-f1618d232941/volumes" Oct 31 01:08:08.937596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:08.937565 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 01:08:08.938047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:08.937778 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" podUID="4836cb85-6281-4d15-85ff-ffdea2e13afe" containerName="maas-api" containerID="cri-o://83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d" gracePeriod=30 Oct 31 01:08:09.411635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.411597 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 01:08:09.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.495334 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\") pod \"4836cb85-6281-4d15-85ff-ffdea2e13afe\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " Oct 31 01:08:09.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.495435 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\") pod \"4836cb85-6281-4d15-85ff-ffdea2e13afe\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " Oct 31 01:08:09.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.495477 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\") pod \"4836cb85-6281-4d15-85ff-ffdea2e13afe\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " Oct 31 01:08:09.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.495516 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\" (UniqueName: \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\") pod \"4836cb85-6281-4d15-85ff-ffdea2e13afe\" (UID: \"4836cb85-6281-4d15-85ff-ffdea2e13afe\") " Oct 31 01:08:09.497277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.496592 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca" pod "4836cb85-6281-4d15-85ff-ffdea2e13afe" (UID: "4836cb85-6281-4d15-85ff-ffdea2e13afe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:09.499432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.499397 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs" pod "4836cb85-6281-4d15-85ff-ffdea2e13afe" (UID: "4836cb85-6281-4d15-85ff-ffdea2e13afe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:09.499571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.499497 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw" pod "4836cb85-6281-4d15-85ff-ffdea2e13afe" (UID: "4836cb85-6281-4d15-85ff-ffdea2e13afe"). InnerVolumeSpecName "kube-api-access-d6shw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:09.499651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.499629 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls" pod "4836cb85-6281-4d15-85ff-ffdea2e13afe" (UID: "4836cb85-6281-4d15-85ff-ffdea2e13afe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:09.597001 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.596962 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4836cb85-6281-4d15-85ff-ffdea2e13afe-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:09.597001 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.596996 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:09.597001 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.597006 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6shw\" (UniqueName: \"kubernetes.io/projected/4836cb85-6281-4d15-85ff-ffdea2e13afe-kube-api-access-d6shw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:09.597331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.597016 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4836cb85-6281-4d15-85ff-ffdea2e13afe-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:09.638461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.638416 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49"] Oct 31 01:08:09.641953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.641925 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c942c9-wwr49"] Oct 31 01:08:09.685832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.685796 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d8ed87-9dc1-48fc-ac60-c15c35a818c5" path="/var/lib/kubelet/pods/c0d8ed87-9dc1-48fc-ac60-c15c35a818c5/volumes" Oct 31 01:08:09.703903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.703875 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:09.704008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.703943 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:09.725075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.725017 5011 generic.go:356] "Generic (PLEG): container finished" podID="261e6357-4d20-4a20-be52-3e0f630fed32" containerID="06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4" exitCode=0 Oct 31 01:08:09.725373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.725348 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerDied","Data":"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4"} Oct 31 01:08:09.725883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.725861 5011 scope.go:122] "RemoveContainer" containerID="06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4" Oct 31 01:08:09.728592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.728555 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:09.728692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.728649 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:09.728739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.728695 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:09.729502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.729474 5011 generic.go:356] "Generic (PLEG): container finished" podID="4836cb85-6281-4d15-85ff-ffdea2e13afe" containerID="83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d" exitCode=0 Oct 31 01:08:09.729622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.729567 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" event={"ID":"4836cb85-6281-4d15-85ff-ffdea2e13afe","Type":"ContainerDied","Data":"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d"} Oct 31 01:08:09.729727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.729622 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" Oct 31 01:08:09.729727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.729637 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5" event={"ID":"4836cb85-6281-4d15-85ff-ffdea2e13afe","Type":"ContainerDied","Data":"8e676d7732de3010b42b12b3b8971629121025f97684c345f8b7b586af68564e"} Oct 31 01:08:09.729727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.729660 5011 scope.go:122] "RemoveContainer" containerID="83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d" Oct 31 01:08:09.747166 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.747143 5011 scope.go:122] "RemoveContainer" containerID="83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d" Oct 31 01:08:09.747531 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:08:09.747484 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d\": container with ID starting with 83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d not found: ID does not exist" containerID="83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d" Oct 31 01:08:09.747531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.747517 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d"} err="failed to get container status \"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d\": rpc error: code = NotFound desc = could not find container \"83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d\": container with ID starting with 83ca2dd64d4075f48b3872e7a76374328f6346ed13cf7798d1ed70443426a04d not found: ID does not exist" Oct 31 01:08:09.817630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.816695 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 01:08:09.821667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:09.821641 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c942c9-64c94c9df8-r2ks5"] Oct 31 01:08:10.756253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:10.753197 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerStarted","Data":"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2"} Oct 31 01:08:11.176914 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:11.176876 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:08:11.177118 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:11.177073 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:11.177190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:11.177126 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:11.684054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:11.684022 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4836cb85-6281-4d15-85ff-ffdea2e13afe" path="/var/lib/kubelet/pods/4836cb85-6281-4d15-85ff-ffdea2e13afe/volumes" Oct 31 01:08:12.176972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:12.176936 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:12.177473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:12.177004 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:13.177389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:13.177355 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:13.177799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:13.177418 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:14.176819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:14.176783 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:14.177052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:14.176847 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:15.176810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:15.176774 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:08:15.177322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:15.176824 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 01:08:16.182770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:16.182737 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:08:16.183275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:16.183239 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:08:16.184038 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:16.184015 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:08:17.383393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.383360 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5"] Oct 31 01:08:17.383957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.383943 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4836cb85-6281-4d15-85ff-ffdea2e13afe" containerName="maas-api" Oct 31 01:08:17.383998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.383958 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4836cb85-6281-4d15-85ff-ffdea2e13afe" containerName="maas-api" Oct 31 01:08:17.384076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.384066 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4836cb85-6281-4d15-85ff-ffdea2e13afe" containerName="maas-api" Oct 31 01:08:17.388818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.388798 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.405911 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.405884 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5"] Oct 31 01:08:17.411912 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.411889 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:17.412031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.411946 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:08:17.442897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.442857 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz9c\" (UniqueName: \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.443073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.443033 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.443170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.443126 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.543952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.543920 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.544121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.544015 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz9c\" (UniqueName: \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.544195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.544139 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.544722 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.544695 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.546644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.546621 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.567873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.567844 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz9c\" (UniqueName: \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\") pod \"maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:17.704244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:17.704134 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:18.019567 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:08:18.019535 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b007e60_d6d8_48aa_a866_aa6d16564471.slice/crio-6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc WatchSource:0}: Error finding container 6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc: Status 404 returned error can't find the container with id 6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc Oct 31 01:08:18.020427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:18.020365 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5"] Oct 31 01:08:18.825897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:18.825856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" event={"ID":"4b007e60-d6d8-48aa-a866-aa6d16564471","Type":"ContainerStarted","Data":"c7e4dccda2c51fb0475628a189781aa7766a146bc668829843c83ce02ca2f13d"} Oct 31 01:08:18.826395 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:18.825936 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" event={"ID":"4b007e60-d6d8-48aa-a866-aa6d16564471","Type":"ContainerStarted","Data":"6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc"} Oct 31 01:08:18.869604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:18.869553 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" podStartSLOduration=1.869539869 podStartE2EDuration="1.869539869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:18.866467357 +0000 UTC m=+2967.315554997" watchObservedRunningTime="2026-10-31 01:08:18.869539869 +0000 UTC m=+2967.318627512" Oct 31 01:08:19.837903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:19.837865 5011 generic.go:356] "Generic (PLEG): container finished" podID="4b007e60-d6d8-48aa-a866-aa6d16564471" containerID="c7e4dccda2c51fb0475628a189781aa7766a146bc668829843c83ce02ca2f13d" exitCode=0 Oct 31 01:08:19.838309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:19.837921 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" event={"ID":"4b007e60-d6d8-48aa-a866-aa6d16564471","Type":"ContainerDied","Data":"c7e4dccda2c51fb0475628a189781aa7766a146bc668829843c83ce02ca2f13d"} Oct 31 01:08:21.002091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.002066 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:21.091604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.091573 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\") pod \"4b007e60-d6d8-48aa-a866-aa6d16564471\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " Oct 31 01:08:21.091759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.091627 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\" (UniqueName: \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\") pod \"4b007e60-d6d8-48aa-a866-aa6d16564471\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " Oct 31 01:08:21.091759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.091709 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\") pod \"4b007e60-d6d8-48aa-a866-aa6d16564471\" (UID: \"4b007e60-d6d8-48aa-a866-aa6d16564471\") " Oct 31 01:08:21.092070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.092039 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca" pod "4b007e60-d6d8-48aa-a866-aa6d16564471" (UID: "4b007e60-d6d8-48aa-a866-aa6d16564471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:21.092309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.092287 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:21.093903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.093877 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca" pod "4b007e60-d6d8-48aa-a866-aa6d16564471" (UID: "4b007e60-d6d8-48aa-a866-aa6d16564471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:21.094088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.094064 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c" pod "4b007e60-d6d8-48aa-a866-aa6d16564471" (UID: "4b007e60-d6d8-48aa-a866-aa6d16564471"). InnerVolumeSpecName "kube-api-access-jrz9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:21.193417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.193386 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b007e60-d6d8-48aa-a866-aa6d16564471-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:21.193417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.193416 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrz9c\" (UniqueName: \"kubernetes.io/projected/4b007e60-d6d8-48aa-a866-aa6d16564471-kube-api-access-jrz9c\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:08:21.856354 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.856320 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" Oct 31 01:08:21.856354 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.856333 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5" event={"ID":"4b007e60-d6d8-48aa-a866-aa6d16564471","Type":"ContainerDied","Data":"6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc"} Oct 31 01:08:21.856553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:21.856388 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a55a0fa998e5e2d2a485711263fd9d58d15ca35fc0f2c0aab7c79d5ee81f4bc" Oct 31 01:08:30.019479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.019412 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:08:30.020123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.020029 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4b007e60-d6d8-48aa-a866-aa6d16564471" containerName="revoke-keys" Oct 31 01:08:30.020123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.020044 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b007e60-d6d8-48aa-a866-aa6d16564471" containerName="revoke-keys" Oct 31 01:08:30.020291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.020159 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b007e60-d6d8-48aa-a866-aa6d16564471" containerName="revoke-keys" Oct 31 01:08:30.025403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.025376 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.044553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.044504 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:08:30.056314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.056289 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-bcdb4e\"/\"kube-root-ca.crt\"" Oct 31 01:08:30.058742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.058712 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-bcdb4e\"/\"rate-test-model-bcdb4e-kserve-self-signed-certs\"" Oct 31 01:08:30.060758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.060735 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-bcdb4e\"/\"default-dockercfg-7khjf\"" Oct 31 01:08:30.144489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.144449 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.144953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.144920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kss\" (UniqueName: \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.145128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.145011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.145128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.145056 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.145284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.145268 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.145347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.145321 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.151612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.151555 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-bcdb4e\"/\"openshift-service-ca.crt\"" Oct 31 01:08:30.246596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.246804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246619 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.246804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246748 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.246804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kss\" (UniqueName: \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.246993 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246855 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.246993 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.246895 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.247500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.247467 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.247943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.247919 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.248416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.248387 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.256535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.256463 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.273137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.273061 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kss\" (UniqueName: \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.284251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.274845 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\") pod \"rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.363948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.361470 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:30.687304 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:08:30.687259 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode09d287e_3aad_464b_ba4b_11252ed5b03f.slice/crio-e68d8b6d56b0de5ef087b0e1b17fb79b1b4c619e4385ebd11f6346da523dd5dc WatchSource:0}: Error finding container e68d8b6d56b0de5ef087b0e1b17fb79b1b4c619e4385ebd11f6346da523dd5dc: Status 404 returned error can't find the container with id e68d8b6d56b0de5ef087b0e1b17fb79b1b4c619e4385ebd11f6346da523dd5dc Oct 31 01:08:30.687613 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.687583 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:08:30.936577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.936487 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" event={"ID":"e09d287e-3aad-464b-ba4b-11252ed5b03f","Type":"ContainerStarted","Data":"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293"} Oct 31 01:08:30.936577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.936534 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" event={"ID":"e09d287e-3aad-464b-ba4b-11252ed5b03f","Type":"ContainerStarted","Data":"e68d8b6d56b0de5ef087b0e1b17fb79b1b4c619e4385ebd11f6346da523dd5dc"} Oct 31 01:08:30.979363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:30.979312 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" podStartSLOduration=1.979298774 podStartE2EDuration="1.979298774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:30.977587402 +0000 UTC m=+2979.426675041" watchObservedRunningTime="2026-10-31 01:08:30.979298774 +0000 UTC m=+2979.428386443" Oct 31 01:08:31.362661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:31.362629 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:31.948290 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:08:31.948256 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:08:40.362367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:40.362327 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:40.375862 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:40.375836 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:41.037076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:41.037035 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:08:42.044728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:42.044691 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc"] Oct 31 01:08:42.057110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:42.057080 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83eb2066-q6jkc"] Oct 31 01:08:43.684378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:43.684346 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399" path="/var/lib/kubelet/pods/6a6248d3-1ec4-4a86-a46d-e3b7f0b2b399/volumes" Oct 31 01:08:52.462992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:52.462962 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:08:52.473250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:52.473192 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:08:53.949579 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:53.949546 5011 scope.go:122] "RemoveContainer" containerID="4272c1bddb10f89a410d33026ad0ea79507e04e9fcdae5038b6cbe1fd31f8845" Oct 31 01:08:56.225893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:56.225859 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5"] Oct 31 01:08:56.237838 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:56.237811 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f8856-hvrz5"] Oct 31 01:08:57.683877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:08:57.683841 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b007e60-d6d8-48aa-a866-aa6d16564471" path="/var/lib/kubelet/pods/4b007e60-d6d8-48aa-a866-aa6d16564471/volumes" Oct 31 01:09:01.947810 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:09:01.947768 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:09:10.032617 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:10.032540 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc"] Oct 31 01:09:10.043254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:10.043222 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ebe8-f5bnc"] Oct 31 01:09:11.686408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:11.686375 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af89544b-7560-43d4-987b-ccf30874f3ec" path="/var/lib/kubelet/pods/af89544b-7560-43d4-987b-ccf30874f3ec/volumes" Oct 31 01:09:24.644346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:24.644310 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:09:31.982749 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:09:31.982705 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:09:36.534096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.534011 5011 generic.go:356] "Generic (PLEG): container finished" podID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerID="8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a" exitCode=0 Oct 31 01:09:36.534549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.534083 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerDied","Data":"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a"} Oct 31 01:09:36.534685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.534667 5011 scope.go:122] "RemoveContainer" containerID="8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a" Oct 31 01:09:36.536754 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.536725 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:36.536866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.536796 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:36.536866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:36.536829 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:37.557864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:37.557819 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerStarted","Data":"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9"} Oct 31 01:09:38.048431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:38.048384 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52"] Oct 31 01:09:38.061400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:38.061364 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f180056-s6t52"] Oct 31 01:09:38.241601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:38.241563 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:09:38.241956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:38.241934 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 01:09:38.242228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:38.242088 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 01:09:39.250502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:39.250458 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 01:09:39.251180 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:39.251143 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 01:09:39.685172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:39.685136 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b613dec-b1dc-4273-879c-1401953c6e6e" path="/var/lib/kubelet/pods/8b613dec-b1dc-4273-879c-1401953c6e6e/volumes" Oct 31 01:09:40.241232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:40.240777 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 01:09:40.241232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:40.240837 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 01:09:41.238532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:41.238494 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 01:09:41.239121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:41.239092 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 01:09:42.240550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:42.240512 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 31 01:09:42.241172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:42.241136 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 31 01:09:43.243508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:43.243476 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:09:43.244015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:43.243991 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:09:43.244756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:43.244724 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:09:53.986316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:53.986279 5011 scope.go:122] "RemoveContainer" containerID="6d506cae6895ce02e65bccc7bacb7abbbd9157d9d5265b2bdb3b9779acd49252" Oct 31 01:09:53.999990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:53.999959 5011 scope.go:122] "RemoveContainer" containerID="629844486f20a8465c7cda80ab427ae006cfb1bc5de63d431b5e5a5b1bc381f4" Oct 31 01:09:54.010028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:54.010002 5011 scope.go:122] "RemoveContainer" containerID="c5ff85d1c4373bfbadb4deb350a6cef0bbb4b61f8ef3ee028f6cc7ac621a4367" Oct 31 01:09:57.035502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:57.035462 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq"] Oct 31 01:09:57.045392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:57.045357 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048e64cf-hz9lq"] Oct 31 01:09:57.683901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:09:57.683866 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed25d9b-0eff-4779-8414-92228160f332" path="/var/lib/kubelet/pods/7ed25d9b-0eff-4779-8414-92228160f332/volumes" Oct 31 01:10:01.948652 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:10:01.948591 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:10:31.950613 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:10:31.950558 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:10:43.585065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:43.584987 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:10:52.223872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.223829 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:10:52.228298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.228275 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.247002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.246976 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:10:52.254374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.254345 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f915\"" Oct 31 01:10:52.254509 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.254387 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f915-dockercfg-lklrm\"" Oct 31 01:10:52.325834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.325798 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.326010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.325989 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2mb\" (UniqueName: \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.427528 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.427499 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2mb\" (UniqueName: \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.427689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.427624 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.428319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.428301 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.451749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.451721 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2mb\" (UniqueName: \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\") pod \"payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.541975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.541896 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:52.871834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:52.871803 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:10:52.872959 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:10:52.872932 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f7bc64_4532_4757_b8d6_732b4e6b5a01.slice/crio-5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca WatchSource:0}: Error finding container 5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca: Status 404 returned error can't find the container with id 5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca Oct 31 01:10:53.280941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:53.280854 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" event={"ID":"c4f7bc64-4532-4757-b8d6-732b4e6b5a01","Type":"ContainerStarted","Data":"5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca"} Oct 31 01:10:54.050772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:54.050735 5011 scope.go:122] "RemoveContainer" containerID="f8097b42ef145aef7487907bc7d2f06cd55ca7ca4b665e4d0311f07039cd68dc" Oct 31 01:10:54.062732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:54.062708 5011 scope.go:122] "RemoveContainer" containerID="e9b456b05e84230a43afec543818f1ccb85b0875d4d30c784bcdb41ae700013a" Oct 31 01:10:54.073459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:54.073440 5011 scope.go:122] "RemoveContainer" containerID="bb82d90494a8ee076a09e47befc2e237e759b95e282e43f3ed536d0404def4c6" Oct 31 01:10:54.084303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:54.084276 5011 scope.go:122] "RemoveContainer" containerID="44e3c903891b51d7d26bb99ff6707b92c1deeba758e381adb6b8c095541e2203" Oct 31 01:10:55.297139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:55.297105 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" event={"ID":"c4f7bc64-4532-4757-b8d6-732b4e6b5a01","Type":"ContainerStarted","Data":"063d83286ea6ce662e13a14cc5a4108ec399ac7382a570cf434c544f4ee10d8a"} Oct 31 01:10:55.297659 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:55.297177 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:10:55.340119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:10:55.340061 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" podStartSLOduration=1.993365751 podStartE2EDuration="3.340046239s" totalImagesPullingTime="1.346680488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:55.337824665 +0000 UTC m=+3123.786912304" watchObservedRunningTime="2026-10-31 01:10:55.340046239 +0000 UTC m=+3123.789133877" Oct 31 01:11:01.946938 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:11:01.946890 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:11:06.306896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:06.306864 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:11:07.034197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:07.034162 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw"] Oct 31 01:11:07.046770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:07.046735 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2ba884-w4vvw"] Oct 31 01:11:07.684089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:07.684055 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78017d7-f4c0-4e30-a319-ca7e8bd5ae04" path="/var/lib/kubelet/pods/f78017d7-f4c0-4e30-a319-ca7e8bd5ae04/volumes" Oct 31 01:11:17.198097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.198062 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt"] Oct 31 01:11:17.203450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.203427 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.223299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.223222 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt"] Oct 31 01:11:17.231266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.229982 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:17.231266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.230042 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:11:17.385631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.385586 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.385902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.385877 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.386095 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.386071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.461701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.461619 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6"] Oct 31 01:11:17.465733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.465710 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.485196 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.485166 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6"] Oct 31 01:11:17.487197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.487169 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.487337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.487247 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.487678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.487436 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.488316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.488293 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.490598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.490566 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.544569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.544533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\") pod \"maas-api-revoke-sub-87105eff-lbrxt\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:17.589016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.588975 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7q87\" (UniqueName: \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.589200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.589173 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.589300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.589240 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.690447 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.690414 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.690649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.690464 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.690726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.690692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7q87\" (UniqueName: \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.691306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.691271 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.693030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.693000 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.714499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.714443 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7q87\" (UniqueName: \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\") pod \"maas-api-revoke-sub-5530d91c-95bx6\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.781708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.781679 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:17.821476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:17.821447 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:18.119479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:18.119446 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6"] Oct 31 01:11:18.120630 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:18.120500 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119d4c99_a0dc_4cc6_88a4_3c5ef8adcbd3.slice/crio-740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53 WatchSource:0}: Error finding container 740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53: Status 404 returned error can't find the container with id 740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53 Oct 31 01:11:18.171189 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:18.171154 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aaadbb_9253_4008_bb3f_3f70345f1eef.slice/crio-307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671 WatchSource:0}: Error finding container 307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671: Status 404 returned error can't find the container with id 307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671 Oct 31 01:11:18.172713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:18.172680 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt"] Oct 31 01:11:18.491978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:18.491895 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" event={"ID":"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3","Type":"ContainerStarted","Data":"740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53"} Oct 31 01:11:18.493349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:18.493317 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" event={"ID":"b8aaadbb-9253-4008-bb3f-3f70345f1eef","Type":"ContainerStarted","Data":"307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671"} Oct 31 01:11:19.503106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.503020 5011 generic.go:356] "Generic (PLEG): container finished" podID="119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" containerID="8a7f3a5c446589cf1a6ba75744e39c4dd782b497a8d5996479453d6389a28675" exitCode=0 Oct 31 01:11:19.503583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.503167 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" event={"ID":"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3","Type":"ContainerDied","Data":"8a7f3a5c446589cf1a6ba75744e39c4dd782b497a8d5996479453d6389a28675"} Oct 31 01:11:19.505105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.505075 5011 generic.go:356] "Generic (PLEG): container finished" podID="b8aaadbb-9253-4008-bb3f-3f70345f1eef" containerID="c59397ebf2813776ab0072ebe9bbe876e7bca13a0f54d8b2329c2c5674aceed6" exitCode=0 Oct 31 01:11:19.505260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.505152 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" event={"ID":"b8aaadbb-9253-4008-bb3f-3f70345f1eef","Type":"ContainerDied","Data":"c59397ebf2813776ab0072ebe9bbe876e7bca13a0f54d8b2329c2c5674aceed6"} Oct 31 01:11:19.689203 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.689170 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5"] Oct 31 01:11:19.693366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.693345 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.713375 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.713346 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5"] Oct 31 01:11:19.725895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.725865 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.726027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.725950 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.726094 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.726063 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnzx\" (UniqueName: \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.826941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.826912 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.827109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.826986 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.827109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.827040 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnzx\" (UniqueName: \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.827772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.827742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.829488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.829465 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:19.850459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:19.850434 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnzx\" (UniqueName: \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\") pod \"maas-api-revoke-sub-64ca7cbf-hh7z5\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:20.007717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.007693 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:20.324978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.324949 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5"] Oct 31 01:11:20.325567 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:20.325541 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d7b41_f7ac_43ab_a86c_ec3c060e1f2a.slice/crio-3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32 WatchSource:0}: Error finding container 3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32: Status 404 returned error can't find the container with id 3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32 Oct 31 01:11:20.516983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.516945 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" event={"ID":"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a","Type":"ContainerStarted","Data":"3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32"} Oct 31 01:11:20.693515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.693494 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:20.724501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.724472 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:20.751102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.751071 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\" (UniqueName: \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\") pod \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " Oct 31 01:11:20.751323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.751304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\") pod \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " Oct 31 01:11:20.751681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.751659 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca" pod "b8aaadbb-9253-4008-bb3f-3f70345f1eef" (UID: "b8aaadbb-9253-4008-bb3f-3f70345f1eef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:20.751819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.751712 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\") pod \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " Oct 31 01:11:20.752374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.752047 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\") pod \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\" (UID: \"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3\") " Oct 31 01:11:20.752374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.752140 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\") pod \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " Oct 31 01:11:20.752815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.752586 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\") pod \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\" (UID: \"b8aaadbb-9253-4008-bb3f-3f70345f1eef\") " Oct 31 01:11:20.752815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.752642 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca" pod "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" (UID: "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:20.753545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.753522 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:20.754972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.754945 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:20.755441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.755413 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87" pod "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" (UID: "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3"). InnerVolumeSpecName "kube-api-access-h7q87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:20.757177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.757150 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca" pod "b8aaadbb-9253-4008-bb3f-3f70345f1eef" (UID: "b8aaadbb-9253-4008-bb3f-3f70345f1eef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:20.757389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.757369 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca" pod "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" (UID: "119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:20.757617 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.757578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m" pod "b8aaadbb-9253-4008-bb3f-3f70345f1eef" (UID: "b8aaadbb-9253-4008-bb3f-3f70345f1eef"). InnerVolumeSpecName "kube-api-access-q9j6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:20.856254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.856187 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7q87\" (UniqueName: \"kubernetes.io/projected/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-kube-api-access-h7q87\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:20.856254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.856241 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:20.856254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.856256 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8aaadbb-9253-4008-bb3f-3f70345f1eef-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:20.856459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:20.856271 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/b8aaadbb-9253-4008-bb3f-3f70345f1eef-kube-api-access-q9j6m\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:21.528772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.528678 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" Oct 31 01:11:21.528772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.528701 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt" event={"ID":"b8aaadbb-9253-4008-bb3f-3f70345f1eef","Type":"ContainerDied","Data":"307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671"} Oct 31 01:11:21.528772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.528751 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307d0bcf993aec73b5a055005973edbd8635068dca1fe42fae6e8adaf5663671" Oct 31 01:11:21.530787 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.530743 5011 generic.go:356] "Generic (PLEG): container finished" podID="a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" containerID="aa443b70de06398c575c936d85451f89638e76be78f2159182d19a46b69b68e4" exitCode=0 Oct 31 01:11:21.530905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.530828 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" event={"ID":"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a","Type":"ContainerDied","Data":"aa443b70de06398c575c936d85451f89638e76be78f2159182d19a46b69b68e4"} Oct 31 01:11:21.532613 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.532587 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" event={"ID":"119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3","Type":"ContainerDied","Data":"740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53"} Oct 31 01:11:21.532738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.532621 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6" Oct 31 01:11:21.532738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:21.532629 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="740aa80a6197cab25faade18869f6372b3f165da88dd357a8265dd89d5656d53" Oct 31 01:11:22.712830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.712805 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:22.783223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.783160 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\") pod \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " Oct 31 01:11:22.783383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.783356 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\") pod \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " Oct 31 01:11:22.783480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.783464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\" (UniqueName: \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\") pod \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\" (UID: \"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a\") " Oct 31 01:11:22.783564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.783535 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca" pod "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" (UID: "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:22.784163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.784136 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:22.785457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.785425 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca" pod "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" (UID: "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:22.785588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.785542 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx" pod "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" (UID: "a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a"). InnerVolumeSpecName "kube-api-access-hdnzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:22.885357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.885327 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:22.885357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:22.885352 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnzx\" (UniqueName: \"kubernetes.io/projected/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a-kube-api-access-hdnzx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:23.552286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:23.552251 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" Oct 31 01:11:23.552286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:23.552264 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5" event={"ID":"a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a","Type":"ContainerDied","Data":"3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32"} Oct 31 01:11:23.552506 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:23.552347 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c3700a2d61ac5a1658a0d7c9fe492dda62cbfeeff6ec12ac01ccb2b56696c32" Oct 31 01:11:27.533661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.533620 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx"] Oct 31 01:11:27.534509 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534487 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b8aaadbb-9253-4008-bb3f-3f70345f1eef" containerName="revoke-keys" Oct 31 01:11:27.534575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534511 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8aaadbb-9253-4008-bb3f-3f70345f1eef" containerName="revoke-keys" Oct 31 01:11:27.534575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534569 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" containerName="revoke-keys" Oct 31 01:11:27.534647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534577 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" containerName="revoke-keys" Oct 31 01:11:27.534647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534600 5011 cpu_manager.go:414] "containerMap: removing container" podUID="119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" containerName="revoke-keys" Oct 31 01:11:27.534647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534608 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" containerName="revoke-keys" Oct 31 01:11:27.534766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534754 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" containerName="revoke-keys" Oct 31 01:11:27.534801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534773 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" containerName="revoke-keys" Oct 31 01:11:27.534801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.534790 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8aaadbb-9253-4008-bb3f-3f70345f1eef" containerName="revoke-keys" Oct 31 01:11:27.539077 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.539057 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.553830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.553805 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx"] Oct 31 01:11:27.561349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.561324 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:27.561500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.561381 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:11:27.649341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.649309 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.649598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.649573 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.649660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.649628 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrn6\" (UniqueName: \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.751183 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.751137 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.751409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.751245 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrn6\" (UniqueName: \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.751409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.751360 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.752022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.751990 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.754435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.754408 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.775110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.775085 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrn6\" (UniqueName: \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\") pod \"maas-api-revoke-sub-bbc727f3-fdfrx\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.855438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.855405 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:27.872539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.872514 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft"] Oct 31 01:11:27.876797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.876778 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:27.897865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.897817 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft"] Oct 31 01:11:27.961073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.959190 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:27.961073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.959339 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:27.961073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:27.959450 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74v9\" (UniqueName: \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.060750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.060717 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.060956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.060792 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.060956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.060862 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74v9\" (UniqueName: \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.061587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.061562 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.063629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.063599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.086142 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.086120 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74v9\" (UniqueName: \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\") pod \"maas-api-revoke-sub-8731ca16-m9dft\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.198013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.197984 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx"] Oct 31 01:11:28.198948 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:28.198923 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb693b4_564e_4f0e_99a6_816398c3f05a.slice/crio-6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e WatchSource:0}: Error finding container 6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e: Status 404 returned error can't find the container with id 6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e Oct 31 01:11:28.215327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.215307 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:28.527109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.527083 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft"] Oct 31 01:11:28.528445 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:28.528412 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c48266_ba3f_4121_af44_cee5dcba2d9c.slice/crio-08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b WatchSource:0}: Error finding container 08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b: Status 404 returned error can't find the container with id 08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b Oct 31 01:11:28.609150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.609121 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" event={"ID":"dbb693b4-564e-4f0e-99a6-816398c3f05a","Type":"ContainerStarted","Data":"6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e"} Oct 31 01:11:28.610482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:28.610458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" event={"ID":"35c48266-ba3f-4121-af44-cee5dcba2d9c","Type":"ContainerStarted","Data":"08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b"} Oct 31 01:11:29.620289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:29.619725 5011 generic.go:356] "Generic (PLEG): container finished" podID="dbb693b4-564e-4f0e-99a6-816398c3f05a" containerID="749b69b7a592aa77ee021cd3d3cd4c22902e1be72b28a77f3ff44263cca2e44d" exitCode=0 Oct 31 01:11:29.620289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:29.619864 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" event={"ID":"dbb693b4-564e-4f0e-99a6-816398c3f05a","Type":"ContainerDied","Data":"749b69b7a592aa77ee021cd3d3cd4c22902e1be72b28a77f3ff44263cca2e44d"} Oct 31 01:11:29.621964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:29.621927 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" event={"ID":"35c48266-ba3f-4121-af44-cee5dcba2d9c","Type":"ContainerStarted","Data":"8032363debe9aedafeca914f1f232ca8358dea6eba5cd69e958ef603ee66708d"} Oct 31 01:11:29.706813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:29.706762 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" podStartSLOduration=2.706749331 podStartE2EDuration="2.706749331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:29.703785093 +0000 UTC m=+3158.152872732" watchObservedRunningTime="2026-10-31 01:11:29.706749331 +0000 UTC m=+3158.155836969" Oct 31 01:11:30.043407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.043308 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l"] Oct 31 01:11:30.047723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.047691 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc36b9e-dtl4l"] Oct 31 01:11:30.092226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.092186 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk"] Oct 31 01:11:30.104715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.104689 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.118390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.116496 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk"] Oct 31 01:11:30.188859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.188831 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.189017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.188866 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnw4\" (UniqueName: \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.189017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.188997 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.290530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.290495 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.290716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.290539 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnw4\" (UniqueName: \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.290716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.290645 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.291281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.291257 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.292947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.292922 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.314125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.314070 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnw4\" (UniqueName: \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\") pod \"maas-api-revoke-sub-ac45c7b0-4qpfk\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.443329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.443292 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:30.630859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.630824 5011 generic.go:356] "Generic (PLEG): container finished" podID="35c48266-ba3f-4121-af44-cee5dcba2d9c" containerID="8032363debe9aedafeca914f1f232ca8358dea6eba5cd69e958ef603ee66708d" exitCode=0 Oct 31 01:11:30.631283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.630908 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" event={"ID":"35c48266-ba3f-4121-af44-cee5dcba2d9c","Type":"ContainerDied","Data":"8032363debe9aedafeca914f1f232ca8358dea6eba5cd69e958ef603ee66708d"} Oct 31 01:11:30.760838 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.760809 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk"] Oct 31 01:11:30.769508 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:11:30.769472 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5417427_0f66_4752_9ccb_fdde69426373.slice/crio-0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec WatchSource:0}: Error finding container 0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec: Status 404 returned error can't find the container with id 0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec Oct 31 01:11:30.798696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.798673 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:30.899748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.899675 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\") pod \"dbb693b4-564e-4f0e-99a6-816398c3f05a\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " Oct 31 01:11:30.899889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.899785 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\") pod \"dbb693b4-564e-4f0e-99a6-816398c3f05a\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " Oct 31 01:11:30.899940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.899912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\" (UniqueName: \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\") pod \"dbb693b4-564e-4f0e-99a6-816398c3f05a\" (UID: \"dbb693b4-564e-4f0e-99a6-816398c3f05a\") " Oct 31 01:11:30.900044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.900019 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca" pod "dbb693b4-564e-4f0e-99a6-816398c3f05a" (UID: "dbb693b4-564e-4f0e-99a6-816398c3f05a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:30.900362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.900344 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:30.901834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.901808 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca" pod "dbb693b4-564e-4f0e-99a6-816398c3f05a" (UID: "dbb693b4-564e-4f0e-99a6-816398c3f05a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:30.902052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:30.902033 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6" pod "dbb693b4-564e-4f0e-99a6-816398c3f05a" (UID: "dbb693b4-564e-4f0e-99a6-816398c3f05a"). InnerVolumeSpecName "kube-api-access-sfrn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:31.002131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.002104 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb693b4-564e-4f0e-99a6-816398c3f05a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:31.002284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.002132 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrn6\" (UniqueName: \"kubernetes.io/projected/dbb693b4-564e-4f0e-99a6-816398c3f05a-kube-api-access-sfrn6\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:31.643253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.643217 5011 generic.go:356] "Generic (PLEG): container finished" podID="e5417427-0f66-4752-9ccb-fdde69426373" containerID="4e8dbcff2ddbfb0c8769edcc965c9890ae064bda901e53e0f40d6d5a4d03907d" exitCode=0 Oct 31 01:11:31.643697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.643263 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" event={"ID":"e5417427-0f66-4752-9ccb-fdde69426373","Type":"ContainerDied","Data":"4e8dbcff2ddbfb0c8769edcc965c9890ae064bda901e53e0f40d6d5a4d03907d"} Oct 31 01:11:31.643697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.643326 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" event={"ID":"e5417427-0f66-4752-9ccb-fdde69426373","Type":"ContainerStarted","Data":"0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec"} Oct 31 01:11:31.644995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.644972 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" Oct 31 01:11:31.645160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.645005 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx" event={"ID":"dbb693b4-564e-4f0e-99a6-816398c3f05a","Type":"ContainerDied","Data":"6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e"} Oct 31 01:11:31.645160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.645055 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6476f3498f3daa4171e5bcde278ffb0f77bbbfe3b1faa5d22392808e9a23df2e" Oct 31 01:11:31.684830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.684799 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492abce6-025c-471b-b86e-f6377f3e011d" path="/var/lib/kubelet/pods/492abce6-025c-471b-b86e-f6377f3e011d/volumes" Oct 31 01:11:31.818087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.818059 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:31.914327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.914294 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\") pod \"35c48266-ba3f-4121-af44-cee5dcba2d9c\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " Oct 31 01:11:31.914497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.914344 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\") pod \"35c48266-ba3f-4121-af44-cee5dcba2d9c\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " Oct 31 01:11:31.914728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.914707 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\" (UniqueName: \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\") pod \"35c48266-ba3f-4121-af44-cee5dcba2d9c\" (UID: \"35c48266-ba3f-4121-af44-cee5dcba2d9c\") " Oct 31 01:11:31.914907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.914708 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca" pod "35c48266-ba3f-4121-af44-cee5dcba2d9c" (UID: "35c48266-ba3f-4121-af44-cee5dcba2d9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:31.915270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.915250 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:31.916575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.916555 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca" pod "35c48266-ba3f-4121-af44-cee5dcba2d9c" (UID: "35c48266-ba3f-4121-af44-cee5dcba2d9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:31.916691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:31.916674 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9" pod "35c48266-ba3f-4121-af44-cee5dcba2d9c" (UID: "35c48266-ba3f-4121-af44-cee5dcba2d9c"). InnerVolumeSpecName "kube-api-access-f74v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:31.948242 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:11:31.947852 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:11:32.016954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.016919 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74v9\" (UniqueName: \"kubernetes.io/projected/35c48266-ba3f-4121-af44-cee5dcba2d9c-kube-api-access-f74v9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:32.016954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.016947 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c48266-ba3f-4121-af44-cee5dcba2d9c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:32.655515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.655485 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" Oct 31 01:11:32.655928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.655504 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft" event={"ID":"35c48266-ba3f-4121-af44-cee5dcba2d9c","Type":"ContainerDied","Data":"08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b"} Oct 31 01:11:32.655928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.655560 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b987008926a927080fc4ef02725a25564af2002297f054a13f50a4a03b558b" Oct 31 01:11:32.823071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.823048 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:32.929101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.929013 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\") pod \"e5417427-0f66-4752-9ccb-fdde69426373\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " Oct 31 01:11:32.929315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.929133 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\") pod \"e5417427-0f66-4752-9ccb-fdde69426373\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " Oct 31 01:11:32.929441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.929423 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\" (UniqueName: \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\") pod \"e5417427-0f66-4752-9ccb-fdde69426373\" (UID: \"e5417427-0f66-4752-9ccb-fdde69426373\") " Oct 31 01:11:32.929553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.929530 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca" pod "e5417427-0f66-4752-9ccb-fdde69426373" (UID: "e5417427-0f66-4752-9ccb-fdde69426373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:32.929975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.929949 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:32.931610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.931585 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4" pod "e5417427-0f66-4752-9ccb-fdde69426373" (UID: "e5417427-0f66-4752-9ccb-fdde69426373"). InnerVolumeSpecName "kube-api-access-2lnw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:32.931794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:32.931773 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca" pod "e5417427-0f66-4752-9ccb-fdde69426373" (UID: "e5417427-0f66-4752-9ccb-fdde69426373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:33.032005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:33.031969 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnw4\" (UniqueName: \"kubernetes.io/projected/e5417427-0f66-4752-9ccb-fdde69426373-kube-api-access-2lnw4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:33.032141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:33.032009 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5417427-0f66-4752-9ccb-fdde69426373-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:33.664038 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:33.664005 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" Oct 31 01:11:33.664464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:33.664061 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk" event={"ID":"e5417427-0f66-4752-9ccb-fdde69426373","Type":"ContainerDied","Data":"0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec"} Oct 31 01:11:33.664464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:33.664129 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f42d565c97d2ce973c5497809e53a0f429823839c22fa598cd061b069f2efec" Oct 31 01:11:40.385489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:40.385447 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:11:54.144262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:54.144217 5011 scope.go:122] "RemoveContainer" containerID="8308ab8c6dbc4b261a3649305ff50634d3c7db8fe12f010d7e33f27d39cd8055" Oct 31 01:11:54.155050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:54.155020 5011 scope.go:122] "RemoveContainer" containerID="3e0685da1fb5900df3242b5e4480e2a3ee1825cb2be46c81d7639c597e116f10" Oct 31 01:11:54.166815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:54.166785 5011 scope.go:122] "RemoveContainer" containerID="5105e4c277467a1e28a12c86a8a419611115ded455c9fd17e305919fd7879ef0" Oct 31 01:11:55.408031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.407955 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" podUID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" containerName="main" containerID="cri-o://9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406" gracePeriod=45 Oct 31 01:11:55.710849 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.710820 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:11:55.852241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852174 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.852431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852305 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.852431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852351 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.852564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852439 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.852564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852461 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.852564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852481 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:55.852564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852533 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\" (UniqueName: \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\") pod \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\" (UID: \"d6edc1a0-4145-45ec-9372-f3fa9f52f045\") " Oct 31 01:11:55.853246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852563 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:55.853246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.852725 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:55.853246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.853029 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:55.853246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.853053 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:55.853246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.853070 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:55.854649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.854617 5011 generic.go:356] "Generic (PLEG): container finished" podID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" containerID="9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406" exitCode=0 Oct 31 01:11:55.854762 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.854670 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" Oct 31 01:11:55.854923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.854904 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" event={"ID":"d6edc1a0-4145-45ec-9372-f3fa9f52f045","Type":"ContainerDied","Data":"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406"} Oct 31 01:11:55.855012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.854942 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq" event={"ID":"d6edc1a0-4145-45ec-9372-f3fa9f52f045","Type":"ContainerDied","Data":"3680cc442eeb4581f13b3ae3fa8ec56bac324dd111e80cd4fd6d19cfc18dbf2b"} Oct 31 01:11:55.855012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.854966 5011 scope.go:122] "RemoveContainer" containerID="9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406" Oct 31 01:11:55.855249 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.855202 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "kube-api-access-wkcgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:55.855324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.855269 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:55.856605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.856579 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs" pod "d6edc1a0-4145-45ec-9372-f3fa9f52f045" (UID: "d6edc1a0-4145-45ec-9372-f3fa9f52f045"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:55.887794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.887775 5011 scope.go:122] "RemoveContainer" containerID="9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406" Oct 31 01:11:55.888044 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:11:55.888021 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406\": container with ID starting with 9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406 not found: ID does not exist" containerID="9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406" Oct 31 01:11:55.888097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.888054 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406"} err="failed to get container status \"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406\": rpc error: code = NotFound desc = could not find container \"9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406\": container with ID starting with 9ce6453b445af51e9856457415e9e20fb8a806fc9d9da267e180a26f23ac8406 not found: ID does not exist" Oct 31 01:11:55.955241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.955140 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edc1a0-4145-45ec-9372-f3fa9f52f045-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:55.955241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.955174 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkcgv\" (UniqueName: \"kubernetes.io/projected/d6edc1a0-4145-45ec-9372-f3fa9f52f045-kube-api-access-wkcgv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:55.955241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:55.955188 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edc1a0-4145-45ec-9372-f3fa9f52f045-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:11:56.203971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:56.203938 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:11:56.215919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:56.215850 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/auth-test-model-498580-kserve-5cdd747ccc-jn7kq"] Oct 31 01:11:57.684715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:57.684684 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" path="/var/lib/kubelet/pods/d6edc1a0-4145-45ec-9372-f3fa9f52f045/volumes" Oct 31 01:11:59.983435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:11:59.983395 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:12:01.040266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:01.040226 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7"] Oct 31 01:12:01.050185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:01.050152 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2181a5-w6bx7"] Oct 31 01:12:01.683830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:01.683786 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1696aea6-23c9-46ed-92f9-0bcfa4b08284" path="/var/lib/kubelet/pods/1696aea6-23c9-46ed-92f9-0bcfa4b08284/volumes" Oct 31 01:12:01.959251 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:12:01.959157 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:12:03.483429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:03.483353 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:12:09.039675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.039640 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:12:09.908795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.908746 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj"] Oct 31 01:12:09.909600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909575 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" containerName="main" Oct 31 01:12:09.909600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909600 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" containerName="main" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909632 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e5417427-0f66-4752-9ccb-fdde69426373" containerName="revoke-keys" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909641 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5417427-0f66-4752-9ccb-fdde69426373" containerName="revoke-keys" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909685 5011 cpu_manager.go:414] "containerMap: removing container" podUID="dbb693b4-564e-4f0e-99a6-816398c3f05a" containerName="revoke-keys" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909693 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb693b4-564e-4f0e-99a6-816398c3f05a" containerName="revoke-keys" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909708 5011 cpu_manager.go:414] "containerMap: removing container" podUID="35c48266-ba3f-4121-af44-cee5dcba2d9c" containerName="revoke-keys" Oct 31 01:12:09.909763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909715 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c48266-ba3f-4121-af44-cee5dcba2d9c" containerName="revoke-keys" Oct 31 01:12:09.909964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909878 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb693b4-564e-4f0e-99a6-816398c3f05a" containerName="revoke-keys" Oct 31 01:12:09.909964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909894 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c48266-ba3f-4121-af44-cee5dcba2d9c" containerName="revoke-keys" Oct 31 01:12:09.909964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909907 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6edc1a0-4145-45ec-9372-f3fa9f52f045" containerName="main" Oct 31 01:12:09.909964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.909919 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5417427-0f66-4752-9ccb-fdde69426373" containerName="revoke-keys" Oct 31 01:12:09.915669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.915648 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:09.943364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.943331 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:12:09.943364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.943331 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:12:09.944938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.944901 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj"] Oct 31 01:12:09.984071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.984025 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8nh\" (UniqueName: \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:09.984384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.984361 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:09.984531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:09.984514 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.085945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.085908 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.086405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.086012 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.086405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.086066 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8nh\" (UniqueName: \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.086795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.086772 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.088338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.088320 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.114809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.114778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8nh\" (UniqueName: \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\") pod \"maas-api-revoke-sub-5b8f34bb-6pvzj\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.249747 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.249659 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:10.574703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:10.574651 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj"] Oct 31 01:12:10.575194 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:12:10.575165 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a5fa07_5fbc_416e_b948_2aeea815b482.slice/crio-4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa WatchSource:0}: Error finding container 4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa: Status 404 returned error can't find the container with id 4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa Oct 31 01:12:11.010592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:11.010543 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" event={"ID":"25a5fa07-5fbc-416e-b948-2aeea815b482","Type":"ContainerStarted","Data":"a02390dfccc506eff08d5d9a7b0bf02cf487c4cf16594c1e0144504474908bcf"} Oct 31 01:12:11.010762 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:11.010612 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" event={"ID":"25a5fa07-5fbc-416e-b948-2aeea815b482","Type":"ContainerStarted","Data":"4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa"} Oct 31 01:12:11.053029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:11.052974 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" podStartSLOduration=2.052960466 podStartE2EDuration="2.052960466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:11.051469599 +0000 UTC m=+3199.500557239" watchObservedRunningTime="2026-10-31 01:12:11.052960466 +0000 UTC m=+3199.502048105" Oct 31 01:12:12.020781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:12.020742 5011 generic.go:356] "Generic (PLEG): container finished" podID="25a5fa07-5fbc-416e-b948-2aeea815b482" containerID="a02390dfccc506eff08d5d9a7b0bf02cf487c4cf16594c1e0144504474908bcf" exitCode=0 Oct 31 01:12:12.021253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:12.020831 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" event={"ID":"25a5fa07-5fbc-416e-b948-2aeea815b482","Type":"ContainerDied","Data":"a02390dfccc506eff08d5d9a7b0bf02cf487c4cf16594c1e0144504474908bcf"} Oct 31 01:12:13.234693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.234664 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:13.331646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.331609 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\") pod \"25a5fa07-5fbc-416e-b948-2aeea815b482\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " Oct 31 01:12:13.331845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.331827 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\" (UniqueName: \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\") pod \"25a5fa07-5fbc-416e-b948-2aeea815b482\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " Oct 31 01:12:13.331924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.331892 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\") pod \"25a5fa07-5fbc-416e-b948-2aeea815b482\" (UID: \"25a5fa07-5fbc-416e-b948-2aeea815b482\") " Oct 31 01:12:13.331994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.331946 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca" pod "25a5fa07-5fbc-416e-b948-2aeea815b482" (UID: "25a5fa07-5fbc-416e-b948-2aeea815b482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:13.332412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.332387 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:13.334067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.334042 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca" pod "25a5fa07-5fbc-416e-b948-2aeea815b482" (UID: "25a5fa07-5fbc-416e-b948-2aeea815b482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:13.334218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.334192 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh" pod "25a5fa07-5fbc-416e-b948-2aeea815b482" (UID: "25a5fa07-5fbc-416e-b948-2aeea815b482"). InnerVolumeSpecName "kube-api-access-hb8nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:13.434055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.434025 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8nh\" (UniqueName: \"kubernetes.io/projected/25a5fa07-5fbc-416e-b948-2aeea815b482-kube-api-access-hb8nh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:13.434055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:13.434054 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a5fa07-5fbc-416e-b948-2aeea815b482-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:14.041631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:14.041542 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" Oct 31 01:12:14.041631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:14.041576 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj" event={"ID":"25a5fa07-5fbc-416e-b948-2aeea815b482","Type":"ContainerDied","Data":"4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa"} Oct 31 01:12:14.041866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:14.041631 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcd68435f39ccb14e89f40a913e9456d5a59f8f9b205d3c3f3be8fbe05ebbaa" Oct 31 01:12:15.006135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.006089 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" podUID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" containerName="main" containerID="cri-o://089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0" gracePeriod=45 Oct 31 01:12:15.294634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.294600 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:12:15.359736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.359703 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\" (UniqueName: \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.359927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.359763 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.359927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.359919 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.360047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.360010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.360106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.360059 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\" (UniqueName: \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.360106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.360098 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\") pod \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\" (UID: \"91a4ffcd-25f8-4f6a-b399-6d50086a7978\") " Oct 31 01:12:15.360897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.360872 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:15.361287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.361266 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:15.361771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.361751 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:15.363572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.363189 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:15.363934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.363909 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:15.365094 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.365061 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7" pod "91a4ffcd-25f8-4f6a-b399-6d50086a7978" (UID: "91a4ffcd-25f8-4f6a-b399-6d50086a7978"). InnerVolumeSpecName "kube-api-access-n74k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:15.461976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.461929 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:15.461976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.461971 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:15.461976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.461984 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a4ffcd-25f8-4f6a-b399-6d50086a7978-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:15.462189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.461999 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:15.462189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.462012 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n74k7\" (UniqueName: \"kubernetes.io/projected/91a4ffcd-25f8-4f6a-b399-6d50086a7978-kube-api-access-n74k7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:15.462189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:15.462023 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a4ffcd-25f8-4f6a-b399-6d50086a7978-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:16.059759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.059671 5011 generic.go:356] "Generic (PLEG): container finished" podID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" containerID="089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0" exitCode=0 Oct 31 01:12:16.059759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.059721 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" Oct 31 01:12:16.060299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.059750 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" event={"ID":"91a4ffcd-25f8-4f6a-b399-6d50086a7978","Type":"ContainerDied","Data":"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0"} Oct 31 01:12:16.060299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.059800 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk" event={"ID":"91a4ffcd-25f8-4f6a-b399-6d50086a7978","Type":"ContainerDied","Data":"0c31d930d2a2ac61f273ceeaa5a29ae6bb8a7711a456c4516cabd0de37b1397c"} Oct 31 01:12:16.060299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.059822 5011 scope.go:122] "RemoveContainer" containerID="089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0" Oct 31 01:12:16.071087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.071062 5011 scope.go:122] "RemoveContainer" containerID="089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0" Oct 31 01:12:16.071382 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:12:16.071365 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0\": container with ID starting with 089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0 not found: ID does not exist" containerID="089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0" Oct 31 01:12:16.071449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.071392 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0"} err="failed to get container status \"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0\": rpc error: code = NotFound desc = could not find container \"089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0\": container with ID starting with 089ab4bbec4f2f1ab87a0a5949d48617f21bbc93f5c0ec193514b655e9e0f7c0 not found: ID does not exist" Oct 31 01:12:16.116509 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.116481 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:12:16.128693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:16.128669 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/auth-test-model-3c90d8-kserve-7c798bdbbf-4d4pk"] Oct 31 01:12:17.692367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:17.692335 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" path="/var/lib/kubelet/pods/91a4ffcd-25f8-4f6a-b399-6d50086a7978/volumes" Oct 31 01:12:20.238123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238092 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz"] Oct 31 01:12:20.238715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238698 5011 cpu_manager.go:414] "containerMap: removing container" podUID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" containerName="main" Oct 31 01:12:20.238715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238716 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" containerName="main" Oct 31 01:12:20.238835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238748 5011 cpu_manager.go:414] "containerMap: removing container" podUID="25a5fa07-5fbc-416e-b948-2aeea815b482" containerName="revoke-keys" Oct 31 01:12:20.238835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238755 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a5fa07-5fbc-416e-b948-2aeea815b482" containerName="revoke-keys" Oct 31 01:12:20.238938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238867 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a4ffcd-25f8-4f6a-b399-6d50086a7978" containerName="main" Oct 31 01:12:20.238938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.238885 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a5fa07-5fbc-416e-b948-2aeea815b482" containerName="revoke-keys" Oct 31 01:12:20.244397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.244373 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.262735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.262701 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz"] Oct 31 01:12:20.270153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.270125 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:12:20.270307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.270162 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:12:20.317824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.317788 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2cf\" (UniqueName: \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.317981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.317850 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.317981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.317928 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.419406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.419367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2cf\" (UniqueName: \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.419406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.419410 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.419661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.419430 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.420069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.420048 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.421900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.421875 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.442964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.442937 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2cf\" (UniqueName: \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\") pod \"maas-api-revoke-sub-36859b20-p6fsz\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.562929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.562836 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:20.877701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:20.877622 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz"] Oct 31 01:12:20.880462 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:12:20.880435 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569cd7b8_fe30_47bc_b063_42a970791cb1.slice/crio-3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571 WatchSource:0}: Error finding container 3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571: Status 404 returned error can't find the container with id 3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571 Oct 31 01:12:21.107832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:21.107775 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" event={"ID":"569cd7b8-fe30-47bc-b063-42a970791cb1","Type":"ContainerStarted","Data":"3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571"} Oct 31 01:12:21.772421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:21.772381 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:12:22.125746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.125708 5011 generic.go:356] "Generic (PLEG): container finished" podID="569cd7b8-fe30-47bc-b063-42a970791cb1" containerID="b13b57ab775f824f884f412357de143f83c6b5ace799513b24902df435ebb7f8" exitCode=0 Oct 31 01:12:22.125947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.125809 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" event={"ID":"569cd7b8-fe30-47bc-b063-42a970791cb1","Type":"ContainerDied","Data":"b13b57ab775f824f884f412357de143f83c6b5ace799513b24902df435ebb7f8"} Oct 31 01:12:22.901388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.901347 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:12:22.922536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.922499 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:22.923932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.923728 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:12:22.948929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:22.948169 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e41918-openshift-default-dockercfg-qrpf7\"" Oct 31 01:12:23.058858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.058805 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.058887 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.058932 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs4tv\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059022 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059199 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059311 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.059513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.059437 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161398 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161608 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161726 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs4tv\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161793 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.161924 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.162003 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.162593 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.163260 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.163917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.163541 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.167965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.166439 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.167965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.166733 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.177992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.177964 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.209719 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.209646 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.217608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.217582 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs4tv\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.222807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.222576 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\") pod \"e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.270636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.270341 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:23.457647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.457614 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:23.578983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.578947 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\") pod \"569cd7b8-fe30-47bc-b063-42a970791cb1\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " Oct 31 01:12:23.579170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.578996 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\" (UniqueName: \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\") pod \"569cd7b8-fe30-47bc-b063-42a970791cb1\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " Oct 31 01:12:23.579258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.579181 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\") pod \"569cd7b8-fe30-47bc-b063-42a970791cb1\" (UID: \"569cd7b8-fe30-47bc-b063-42a970791cb1\") " Oct 31 01:12:23.579676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.579616 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca" pod "569cd7b8-fe30-47bc-b063-42a970791cb1" (UID: "569cd7b8-fe30-47bc-b063-42a970791cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:23.579901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.579857 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:23.581746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.581713 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf" pod "569cd7b8-fe30-47bc-b063-42a970791cb1" (UID: "569cd7b8-fe30-47bc-b063-42a970791cb1"). InnerVolumeSpecName "kube-api-access-vf2cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:23.581926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.581905 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca" pod "569cd7b8-fe30-47bc-b063-42a970791cb1" (UID: "569cd7b8-fe30-47bc-b063-42a970791cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:23.648157 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:12:23.648118 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dba605_8432_495d_a997_50c8b6338fa1.slice/crio-3f96c6692a3d2915b72aee8c9e8110346eb6ab0ce1e73d5085e511e6e5571b08 WatchSource:0}: Error finding container 3f96c6692a3d2915b72aee8c9e8110346eb6ab0ce1e73d5085e511e6e5571b08: Status 404 returned error can't find the container with id 3f96c6692a3d2915b72aee8c9e8110346eb6ab0ce1e73d5085e511e6e5571b08 Oct 31 01:12:23.648655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.648594 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:12:23.652118 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.652078 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:23.652219 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.652161 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:23.652268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.652201 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:23.680842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.680821 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569cd7b8-fe30-47bc-b063-42a970791cb1-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:23.680842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:23.680842 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2cf\" (UniqueName: \"kubernetes.io/projected/569cd7b8-fe30-47bc-b063-42a970791cb1-kube-api-access-vf2cf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.065279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.064777 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" podUID="b05376bb-68b1-4ace-9729-3d3b557ee305" containerName="main" containerID="cri-o://c8babab1d9d95a141e9c26cf0c44c8ef4be428ef5638b4095b6dc87e20cf8a89" gracePeriod=45 Oct 31 01:12:24.225722 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.225687 5011 generic.go:356] "Generic (PLEG): container finished" podID="b05376bb-68b1-4ace-9729-3d3b557ee305" containerID="c8babab1d9d95a141e9c26cf0c44c8ef4be428ef5638b4095b6dc87e20cf8a89" exitCode=0 Oct 31 01:12:24.225920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.225900 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" event={"ID":"b05376bb-68b1-4ace-9729-3d3b557ee305","Type":"ContainerDied","Data":"c8babab1d9d95a141e9c26cf0c44c8ef4be428ef5638b4095b6dc87e20cf8a89"} Oct 31 01:12:24.227803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.227773 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" event={"ID":"569cd7b8-fe30-47bc-b063-42a970791cb1","Type":"ContainerDied","Data":"3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571"} Oct 31 01:12:24.227865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.227831 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3405c4e0208b934b86f47647f67d14696b42335822e421a422377c7893db8571" Oct 31 01:12:24.227918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.227906 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz" Oct 31 01:12:24.230473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.230443 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerStarted","Data":"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b"} Oct 31 01:12:24.230596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.230497 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerStarted","Data":"3f96c6692a3d2915b72aee8c9e8110346eb6ab0ce1e73d5085e511e6e5571b08"} Oct 31 01:12:24.272895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.272272 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:24.284202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.284168 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:24.285852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.285809 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podStartSLOduration=2.285795184 podStartE2EDuration="2.285795184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:24.283049476 +0000 UTC m=+3212.732137116" watchObservedRunningTime="2026-10-31 01:12:24.285795184 +0000 UTC m=+3212.734882823" Oct 31 01:12:24.371701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.371672 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:12:24.396252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.392890 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\" (UniqueName: \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.396252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.392944 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.396523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.396437 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.396523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.396504 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\" (UniqueName: \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.396638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.396626 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.396738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.396705 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\") pod \"b05376bb-68b1-4ace-9729-3d3b557ee305\" (UID: \"b05376bb-68b1-4ace-9729-3d3b557ee305\") " Oct 31 01:12:24.399707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.397751 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:24.402408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.402362 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:24.403540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.402806 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:24.410682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.410533 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "kube-api-access-nzbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:24.410682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.410635 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:24.421365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.421324 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir" pod "b05376bb-68b1-4ace-9729-3d3b557ee305" (UID: "b05376bb-68b1-4ace-9729-3d3b557ee305"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:24.499338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499303 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbj7\" (UniqueName: \"kubernetes.io/projected/b05376bb-68b1-4ace-9729-3d3b557ee305-kube-api-access-nzbj7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.499338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499333 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.499338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499342 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.499571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499350 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05376bb-68b1-4ace-9729-3d3b557ee305-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.499571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499359 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:24.499571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:24.499367 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05376bb-68b1-4ace-9729-3d3b557ee305-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:25.034255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.033848 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm"] Oct 31 01:12:25.039689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.039654 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05222b04-tc6gm"] Oct 31 01:12:25.240871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.240837 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" Oct 31 01:12:25.241511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.240844 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk" event={"ID":"b05376bb-68b1-4ace-9729-3d3b557ee305","Type":"ContainerDied","Data":"ddeba1009cc600f582d4cddd28ddae600d815c28e23e0974856d5ac417e98c49"} Oct 31 01:12:25.241511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.240968 5011 scope.go:122] "RemoveContainer" containerID="c8babab1d9d95a141e9c26cf0c44c8ef4be428ef5638b4095b6dc87e20cf8a89" Oct 31 01:12:25.241805 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.241669 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:25.245856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.245807 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:12:25.294064 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.294005 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:12:25.297709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.297688 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-476a67/auth-test-model-476a67-kserve-65c76885d9-hc9tk"] Oct 31 01:12:25.684512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.684481 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05376bb-68b1-4ace-9729-3d3b557ee305" path="/var/lib/kubelet/pods/b05376bb-68b1-4ace-9729-3d3b557ee305/volumes" Oct 31 01:12:25.684974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:25.684959 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df529f99-93cc-425e-8f57-19669e2ddee5" path="/var/lib/kubelet/pods/df529f99-93cc-425e-8f57-19669e2ddee5/volumes" Oct 31 01:12:31.948933 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:12:31.948898 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:12:32.252147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252071 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:12:32.252670 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252656 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b05376bb-68b1-4ace-9729-3d3b557ee305" containerName="main" Oct 31 01:12:32.252716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252671 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05376bb-68b1-4ace-9729-3d3b557ee305" containerName="main" Oct 31 01:12:32.252716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252686 5011 cpu_manager.go:414] "containerMap: removing container" podUID="569cd7b8-fe30-47bc-b063-42a970791cb1" containerName="revoke-keys" Oct 31 01:12:32.252716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252691 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569cd7b8-fe30-47bc-b063-42a970791cb1" containerName="revoke-keys" Oct 31 01:12:32.252821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252791 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05376bb-68b1-4ace-9729-3d3b557ee305" containerName="main" Oct 31 01:12:32.252821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.252800 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="569cd7b8-fe30-47bc-b063-42a970791cb1" containerName="revoke-keys" Oct 31 01:12:32.259578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.259542 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.281185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.281158 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:12:32.285500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.285470 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"kube-root-ca.crt\"" Oct 31 01:12:32.286415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.286382 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"openshift-service-ca.crt\"" Oct 31 01:12:32.286541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.286384 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"default-dockercfg-ll2nn\"" Oct 31 01:12:32.286541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.286386 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-498580\"/\"sub-test-model-498580-kserve-self-signed-certs\"" Oct 31 01:12:32.323499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.323468 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.323687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.323522 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.323687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.323662 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.324193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.323736 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.324193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.323977 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.324193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.324014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64thh\" (UniqueName: \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425291 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64thh\" (UniqueName: \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425343 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425383 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425660 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425775 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.425994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.425974 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.426384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.426353 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.427827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.427802 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.428360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.428342 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.450016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.449989 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64thh\" (UniqueName: \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\") pod \"sub-test-model-498580-kserve-656c554fb9-kwt4z\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.581950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.581914 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:32.925808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:32.925780 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:12:32.926382 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:12:32.926357 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13342db5_6125_459d_a545_346edfb59fda.slice/crio-ba0b100e9811d13a5a6a3cc5f89e89d731bba77bf438c5019dc352065dabcf1d WatchSource:0}: Error finding container ba0b100e9811d13a5a6a3cc5f89e89d731bba77bf438c5019dc352065dabcf1d: Status 404 returned error can't find the container with id ba0b100e9811d13a5a6a3cc5f89e89d731bba77bf438c5019dc352065dabcf1d Oct 31 01:12:33.308989 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:33.308902 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" event={"ID":"13342db5-6125-459d-a545-346edfb59fda","Type":"ContainerStarted","Data":"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411"} Oct 31 01:12:33.308989 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:33.308956 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" event={"ID":"13342db5-6125-459d-a545-346edfb59fda","Type":"ContainerStarted","Data":"ba0b100e9811d13a5a6a3cc5f89e89d731bba77bf438c5019dc352065dabcf1d"} Oct 31 01:12:33.362746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:33.362680 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" podStartSLOduration=1.362662905 podStartE2EDuration="1.362662905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:33.359964968 +0000 UTC m=+3221.809052606" watchObservedRunningTime="2026-10-31 01:12:33.362662905 +0000 UTC m=+3221.811750561" Oct 31 01:12:33.582810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:33.582771 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:34.607229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.607180 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:12:34.612843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.612817 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.634824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.634794 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:12:34.640660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.640629 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"openshift-service-ca.crt\"" Oct 31 01:12:34.641074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.640692 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"sub-test-model-36af31-kserve-self-signed-certs\"" Oct 31 01:12:34.641292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.640823 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"default-dockercfg-hz9tr\"" Oct 31 01:12:34.641292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.640918 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-36af31\"/\"kube-root-ca.crt\"" Oct 31 01:12:34.657998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.657968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.658237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.658218 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.658320 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.658292 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.658395 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.658359 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.658435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.658414 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.658482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.658453 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzrz\" (UniqueName: \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.759989 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760035 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760058 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzrz\" (UniqueName: \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760269 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760459 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760503 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760514 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.760695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760548 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.761521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.760946 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.763268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.763244 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.763380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.763322 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.786256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.786227 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzrz\" (UniqueName: \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\") pod \"sub-test-model-36af31-kserve-65c9674dd5-mvmds\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:34.939880 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:34.939795 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:35.283377 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:12:35.283339 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8969133a_2929_4873_9721_f7bae064c00d.slice/crio-e9ff2379c64c0681bda824bbff7654938828bb1f4a91ac3a5dba61c9a77e7a5d WatchSource:0}: Error finding container e9ff2379c64c0681bda824bbff7654938828bb1f4a91ac3a5dba61c9a77e7a5d: Status 404 returned error can't find the container with id e9ff2379c64c0681bda824bbff7654938828bb1f4a91ac3a5dba61c9a77e7a5d Oct 31 01:12:35.285538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:35.285450 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:12:35.328818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:35.328785 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" event={"ID":"8969133a-2929-4873-9721-f7bae064c00d","Type":"ContainerStarted","Data":"e9ff2379c64c0681bda824bbff7654938828bb1f4a91ac3a5dba61c9a77e7a5d"} Oct 31 01:12:36.338707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:36.338664 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" event={"ID":"8969133a-2929-4873-9721-f7bae064c00d","Type":"ContainerStarted","Data":"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179"} Oct 31 01:12:36.383609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:36.383557 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" podStartSLOduration=2.383543867 podStartE2EDuration="2.383543867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:36.379738973 +0000 UTC m=+3224.828826611" watchObservedRunningTime="2026-10-31 01:12:36.383543867 +0000 UTC m=+3224.832631500" Oct 31 01:12:36.798851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:36.798762 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" podUID="48733644-b6af-40f5-8810-ddff6a976c8a" containerName="main" containerID="cri-o://19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff" gracePeriod=45 Oct 31 01:12:36.940126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:36.940097 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:37.095459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.095434 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:12:37.202707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202668 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\" (UniqueName: \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.202898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202770 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.202898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202818 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.202898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.202898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202892 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.203182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.202958 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\") pod \"48733644-b6af-40f5-8810-ddff6a976c8a\" (UID: \"48733644-b6af-40f5-8810-ddff6a976c8a\") " Oct 31 01:12:37.203182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203039 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:37.203323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203172 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:37.203388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203336 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:37.203649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203629 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.203732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203655 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.203732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.203670 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.205200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.205177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:37.205333 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.205286 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:37.205394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.205377 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq" pod "48733644-b6af-40f5-8810-ddff6a976c8a" (UID: "48733644-b6af-40f5-8810-ddff6a976c8a"). InnerVolumeSpecName "kube-api-access-zxgpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:37.304876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.304842 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48733644-b6af-40f5-8810-ddff6a976c8a-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.304876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.304871 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48733644-b6af-40f5-8810-ddff6a976c8a-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.304876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.304883 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgpq\" (UniqueName: \"kubernetes.io/projected/48733644-b6af-40f5-8810-ddff6a976c8a-kube-api-access-zxgpq\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:37.347838 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.347804 5011 generic.go:356] "Generic (PLEG): container finished" podID="48733644-b6af-40f5-8810-ddff6a976c8a" containerID="19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff" exitCode=0 Oct 31 01:12:37.348276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.347849 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" Oct 31 01:12:37.348276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.347892 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" event={"ID":"48733644-b6af-40f5-8810-ddff6a976c8a","Type":"ContainerDied","Data":"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff"} Oct 31 01:12:37.348276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.347939 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f" event={"ID":"48733644-b6af-40f5-8810-ddff6a976c8a","Type":"ContainerDied","Data":"3257e580ffbbcad45832fd0c6859b724ead51963e405d97138a958efadf2e973"} Oct 31 01:12:37.348276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.347962 5011 scope.go:122] "RemoveContainer" containerID="19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff" Oct 31 01:12:37.360331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.360309 5011 scope.go:122] "RemoveContainer" containerID="19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff" Oct 31 01:12:37.360618 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:12:37.360598 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff\": container with ID starting with 19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff not found: ID does not exist" containerID="19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff" Oct 31 01:12:37.360696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.360625 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff"} err="failed to get container status \"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff\": rpc error: code = NotFound desc = could not find container \"19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff\": container with ID starting with 19add79e95a6c135624ed9ad52e56bcfe1c4318e4c4d9bf3362d63a25cf69cff not found: ID does not exist" Oct 31 01:12:37.408154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.408127 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:12:37.411798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.411775 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-b8fd5a/auth-test-model-b8fd5a-kserve-6c94c96d9c-q498f"] Oct 31 01:12:37.683580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:37.683505 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48733644-b6af-40f5-8810-ddff6a976c8a" path="/var/lib/kubelet/pods/48733644-b6af-40f5-8810-ddff6a976c8a/volumes" Oct 31 01:12:42.409132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:42.409095 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:12:42.583232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:42.583178 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:42.692325 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:42.691390 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:43.412221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:43.412177 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:12:44.940005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:44.939964 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:44.957930 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:44.957904 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:45.452148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:45.452113 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:12:54.205410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:54.205376 5011 scope.go:122] "RemoveContainer" containerID="8eeaeb93d1aa5967091cb20fe309b829818ee2ee832a4b796361dcc48bce5582" Oct 31 01:12:54.215825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:54.215799 5011 scope.go:122] "RemoveContainer" containerID="3a9b5459c57e18551bbc1241befd9ad8f3c031e06f18b75a5bef78966b1c435b" Oct 31 01:12:54.227708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:54.227677 5011 scope.go:122] "RemoveContainer" containerID="9dcb7dbf4232f67d7a070ef1fe5d1c06571464cddd080966a3814f0b6aa25cf5" Oct 31 01:12:57.431813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.431738 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" podUID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" containerName="main" containerID="cri-o://5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6" gracePeriod=45 Oct 31 01:12:57.726218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.726173 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:12:57.783154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783123 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783189 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783343 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\" (UniqueName: \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783386 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783421 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:57.783519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783446 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783531 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\" (UniqueName: \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\") pod \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\" (UID: \"05ce1288-cc3e-483e-bdea-8e8eb0036dea\") " Oct 31 01:12:57.783690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783662 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:57.783799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.783700 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:57.785148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.785120 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:57.785314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.785162 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:57.785314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.785182 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:57.786351 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.786321 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:57.786746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.786724 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "kube-api-access-ffvdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:57.786967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.786949 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm" pod "05ce1288-cc3e-483e-bdea-8e8eb0036dea" (UID: "05ce1288-cc3e-483e-bdea-8e8eb0036dea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:57.886340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.886306 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvdr\" (UniqueName: \"kubernetes.io/projected/05ce1288-cc3e-483e-bdea-8e8eb0036dea-kube-api-access-ffvdr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:57.886340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.886335 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ce1288-cc3e-483e-bdea-8e8eb0036dea-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:57.886340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:57.886346 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ce1288-cc3e-483e-bdea-8e8eb0036dea-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:12:58.545051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.545016 5011 generic.go:356] "Generic (PLEG): container finished" podID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" containerID="5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6" exitCode=0 Oct 31 01:12:58.545556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.545085 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" Oct 31 01:12:58.545556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.545092 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" event={"ID":"05ce1288-cc3e-483e-bdea-8e8eb0036dea","Type":"ContainerDied","Data":"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6"} Oct 31 01:12:58.545556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.545200 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt" event={"ID":"05ce1288-cc3e-483e-bdea-8e8eb0036dea","Type":"ContainerDied","Data":"ea9b33584220ec25890514d9ab0d37bdb22a2865822832e963d1e112614ae010"} Oct 31 01:12:58.545556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.545233 5011 scope.go:122] "RemoveContainer" containerID="5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6" Oct 31 01:12:58.565002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.561805 5011 scope.go:122] "RemoveContainer" containerID="5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6" Oct 31 01:12:58.565002 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:12:58.563843 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6\": container with ID starting with 5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6 not found: ID does not exist" containerID="5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6" Oct 31 01:12:58.565002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.563885 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6"} err="failed to get container status \"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6\": rpc error: code = NotFound desc = could not find container \"5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6\": container with ID starting with 5179aff9b7c91e3fee2e2be32e30ab0f80ad7ee396ae576492411f9f36c523d6 not found: ID does not exist" Oct 31 01:12:58.598672 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.598642 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:12:58.610918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:58.610889 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-99a5c2/auth-test-model-99a5c2-kserve-869cdb9cb5-gh8qt"] Oct 31 01:12:59.730798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:12:59.730757 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" path="/var/lib/kubelet/pods/05ce1288-cc3e-483e-bdea-8e8eb0036dea/volumes" Oct 31 01:13:01.948709 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:13:01.948674 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:13:03.481604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.481567 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:13:03.482315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482285 5011 cpu_manager.go:414] "containerMap: removing container" podUID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" containerName="main" Oct 31 01:13:03.482315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482308 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" containerName="main" Oct 31 01:13:03.482512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482366 5011 cpu_manager.go:414] "containerMap: removing container" podUID="48733644-b6af-40f5-8810-ddff6a976c8a" containerName="main" Oct 31 01:13:03.482512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482375 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48733644-b6af-40f5-8810-ddff6a976c8a" containerName="main" Oct 31 01:13:03.482512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482497 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ce1288-cc3e-483e-bdea-8e8eb0036dea" containerName="main" Oct 31 01:13:03.482512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.482514 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="48733644-b6af-40f5-8810-ddff6a976c8a" containerName="main" Oct 31 01:13:03.487651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.487630 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.503790 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.503764 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75ead8\"" Oct 31 01:13:03.505662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.505633 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:13:03.512714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.512691 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75ead8-dockercfg-7bngp\"" Oct 31 01:13:03.609373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.609337 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8swg\" (UniqueName: \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.609555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.609393 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.710848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.710814 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8swg\" (UniqueName: \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.711015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.710872 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.711663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.711641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.734847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.734786 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8swg\" (UniqueName: \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\") pod \"payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:03.807535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:03.807502 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:04.133235 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:13:04.131244 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020d2d90_4a77_4a24_a4ac_52b4ae6f1439.slice/crio-b7cb5d931c136f740ef3c5b77edcc96ab2e324a7bedad2fb6b01342f170aaf05 WatchSource:0}: Error finding container b7cb5d931c136f740ef3c5b77edcc96ab2e324a7bedad2fb6b01342f170aaf05: Status 404 returned error can't find the container with id b7cb5d931c136f740ef3c5b77edcc96ab2e324a7bedad2fb6b01342f170aaf05 Oct 31 01:13:04.133235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:04.132693 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:13:04.609069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:04.609019 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" event={"ID":"020d2d90-4a77-4a24-a4ac-52b4ae6f1439","Type":"ContainerStarted","Data":"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03"} Oct 31 01:13:04.609795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:04.609772 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" event={"ID":"020d2d90-4a77-4a24-a4ac-52b4ae6f1439","Type":"ContainerStarted","Data":"b7cb5d931c136f740ef3c5b77edcc96ab2e324a7bedad2fb6b01342f170aaf05"} Oct 31 01:13:04.609924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:04.609906 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:04.654052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:04.653994 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" podStartSLOduration=1.6539759090000001 podStartE2EDuration="1.653975909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:04.651770442 +0000 UTC m=+3253.100858095" watchObservedRunningTime="2026-10-31 01:13:04.653975909 +0000 UTC m=+3253.103063548" Oct 31 01:13:12.301232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:12.301174 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:13:13.514737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.514693 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:13:13.522822 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.521907 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.542045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.542015 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:13:13.545543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.545508 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4207cd-openshift-default-dockercfg-gsk62\"" Oct 31 01:13:13.562682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.562643 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.563058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.563023 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.563328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.563308 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.563566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.563545 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89r89\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.564063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.564024 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.564262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.564244 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.564938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.564908 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.565038 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.565019 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.565100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.565083 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666420 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666542 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666624 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89r89\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.666945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.666840 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.667194 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.667167 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.669045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.668597 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.669045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.668725 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.669605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.669579 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.669954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.669926 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.681230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.679343 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.712987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.708615 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.727449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.727407 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89r89\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.737166 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.737135 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:13.843798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:13.843540 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:14.194994 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:13:14.194964 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70aafa2_aa36_4ca0_b27e_a1fcb68288e1.slice/crio-f3c771188eacf44e2109fd792ff6b722bca63025ec1fc104bf5e34c752685b8d WatchSource:0}: Error finding container f3c771188eacf44e2109fd792ff6b722bca63025ec1fc104bf5e34c752685b8d: Status 404 returned error can't find the container with id f3c771188eacf44e2109fd792ff6b722bca63025ec1fc104bf5e34c752685b8d Oct 31 01:13:14.195151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.194999 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:13:14.197879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.197848 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:14.197953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.197915 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:14.197953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.197944 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:14.716923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.716885 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerStarted","Data":"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740"} Oct 31 01:13:14.717369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.716949 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerStarted","Data":"f3c771188eacf44e2109fd792ff6b722bca63025ec1fc104bf5e34c752685b8d"} Oct 31 01:13:14.764649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.764587 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podStartSLOduration=1.764568827 podStartE2EDuration="1.764568827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:14.762397212 +0000 UTC m=+3263.211484875" watchObservedRunningTime="2026-10-31 01:13:14.764568827 +0000 UTC m=+3263.213656470" Oct 31 01:13:14.844348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:14.844316 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:15.851082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:15.851048 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:16.626814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:16.626776 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:13:16.738135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:16.738098 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:16.739055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:16.739032 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:13:18.021076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.021038 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:13:18.025525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.025504 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.047023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.046989 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:13:18.054602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.054568 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"openshift-service-ca.crt\"" Oct 31 01:13:18.054751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.054619 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"default-dockercfg-td2h5\"" Oct 31 01:13:18.054751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.054669 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"kube-root-ca.crt\"" Oct 31 01:13:18.055520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.055499 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3c90d8\"/\"sub-test-model-3c90d8-kserve-self-signed-certs\"" Oct 31 01:13:18.152014 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.151972 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.152199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.152108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.152199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.152159 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.152382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.152357 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9fj\" (UniqueName: \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.152473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.152457 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.152550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.152534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256039 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256658 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256736 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256909 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9fj\" (UniqueName: \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.257236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.295329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.256506 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.295700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.295658 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.338278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.332690 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.358716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.354251 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.371850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.371818 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9fj\" (UniqueName: \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.377358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.375571 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\") pod \"sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.642642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.642595 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:18.957726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:18.957692 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:13:18.957875 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:13:18.957772 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4877aa36_7654_4e4e_9f54_ae0dfd5b6393.slice/crio-6dd0a09c3751107bb726f355f98e2327bdaf9d1c98b77f57da91e2fcd82f8fee WatchSource:0}: Error finding container 6dd0a09c3751107bb726f355f98e2327bdaf9d1c98b77f57da91e2fcd82f8fee: Status 404 returned error can't find the container with id 6dd0a09c3751107bb726f355f98e2327bdaf9d1c98b77f57da91e2fcd82f8fee Oct 31 01:13:19.802446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:19.802405 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" event={"ID":"4877aa36-7654-4e4e-9f54-ae0dfd5b6393","Type":"ContainerStarted","Data":"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d"} Oct 31 01:13:19.802940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:19.802456 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" event={"ID":"4877aa36-7654-4e4e-9f54-ae0dfd5b6393","Type":"ContainerStarted","Data":"6dd0a09c3751107bb726f355f98e2327bdaf9d1c98b77f57da91e2fcd82f8fee"} Oct 31 01:13:19.846923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:19.846867 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" podStartSLOduration=2.846852504 podStartE2EDuration="2.846852504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:19.845864964 +0000 UTC m=+3268.294952602" watchObservedRunningTime="2026-10-31 01:13:19.846852504 +0000 UTC m=+3268.295940146" Oct 31 01:13:20.643717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:20.643684 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:24.152224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.152180 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j"] Oct 31 01:13:24.156538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.156510 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.171751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.171699 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:24.183968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.183944 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:13:24.190332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.190306 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j"] Oct 31 01:13:24.272824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.272790 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.272999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.272836 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrhh\" (UniqueName: \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.272999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.272958 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.374659 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.374625 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.374888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.374870 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.374943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.374895 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrhh\" (UniqueName: \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.375284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.375261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.377313 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.377293 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.397924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.397903 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrhh\" (UniqueName: \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.472080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.472009 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:24.784559 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:13:24.784529 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee0ac7c_cc4c_4b1d_90f0_f77a966e61cb.slice/crio-52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6 WatchSource:0}: Error finding container 52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6: Status 404 returned error can't find the container with id 52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6 Oct 31 01:13:24.785899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.785870 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j"] Oct 31 01:13:24.841217 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:24.841176 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" event={"ID":"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb","Type":"ContainerStarted","Data":"52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6"} Oct 31 01:13:25.851123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:25.851091 5011 generic.go:356] "Generic (PLEG): container finished" podID="aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" containerID="055e0e0d606588a499982765d876ba1f8d904b43b09d36aacd7b5d5147cdcf49" exitCode=0 Oct 31 01:13:25.851517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:25.851141 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" event={"ID":"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb","Type":"ContainerDied","Data":"055e0e0d606588a499982765d876ba1f8d904b43b09d36aacd7b5d5147cdcf49"} Oct 31 01:13:27.016621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.016594 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:27.214251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.214140 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\") pod \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " Oct 31 01:13:27.214412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.214329 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\") pod \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " Oct 31 01:13:27.214412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.214373 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\" (UniqueName: \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\") pod \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\" (UID: \"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb\") " Oct 31 01:13:27.214751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.214722 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca" pod "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" (UID: "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:27.214987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.214971 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.216287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.216261 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca" pod "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" (UID: "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:27.216762 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.216747 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh" pod "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" (UID: "aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb"). InnerVolumeSpecName "kube-api-access-ptrhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:27.316148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.316110 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrhh\" (UniqueName: \"kubernetes.io/projected/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-kube-api-access-ptrhh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.316313 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.316151 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.326803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.326772 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerName="main" containerID="cri-o://da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752" gracePeriod=45 Oct 31 01:13:27.345350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.345307 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.224:8000/ready\": dial tcp 10.132.0.224:8000: connect: connection refused" Oct 31 01:13:27.601401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.601376 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:13:27.619130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619097 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619296 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\" (UniqueName: \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619300 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619413 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619462 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:27.619562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619552 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619617 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619585 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\") pod \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\" (UID: \"df5a9000-b16f-4cf0-8571-ecad8628a7c4\") " Oct 31 01:13:27.619737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619714 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:27.619951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.619918 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:27.620250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.620227 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.620359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.620254 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.620359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.620268 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.621774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.621752 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "kube-api-access-v5p7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:27.622280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.622250 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:27.622517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.622493 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm" pod "df5a9000-b16f-4cf0-8571-ecad8628a7c4" (UID: "df5a9000-b16f-4cf0-8571-ecad8628a7c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:27.721335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.721302 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5p7s\" (UniqueName: \"kubernetes.io/projected/df5a9000-b16f-4cf0-8571-ecad8628a7c4-kube-api-access-v5p7s\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.721335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.721331 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5a9000-b16f-4cf0-8571-ecad8628a7c4-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.721500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.721341 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5a9000-b16f-4cf0-8571-ecad8628a7c4-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:27.869368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.869339 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" Oct 31 01:13:27.869561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.869389 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j" event={"ID":"aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb","Type":"ContainerDied","Data":"52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6"} Oct 31 01:13:27.869561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.869450 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52770ec7b80bd1a787f1907f7b426717da88b5f686ccf15891becd7405f9bee6" Oct 31 01:13:27.871011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.870983 5011 generic.go:356] "Generic (PLEG): container finished" podID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerID="da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752" exitCode=0 Oct 31 01:13:27.871163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.871020 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" Oct 31 01:13:27.871163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.871020 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" event={"ID":"df5a9000-b16f-4cf0-8571-ecad8628a7c4","Type":"ContainerDied","Data":"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752"} Oct 31 01:13:27.871310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.871164 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw" event={"ID":"df5a9000-b16f-4cf0-8571-ecad8628a7c4","Type":"ContainerDied","Data":"cbe792ab0400de9f291b4ef4f6501f1ad554e0d9e6d3cac0818819218a842d67"} Oct 31 01:13:27.871310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.871191 5011 scope.go:122] "RemoveContainer" containerID="da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752" Oct 31 01:13:27.881848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.881831 5011 scope.go:122] "RemoveContainer" containerID="da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752" Oct 31 01:13:27.882106 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:13:27.882085 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752\": container with ID starting with da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752 not found: ID does not exist" containerID="da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752" Oct 31 01:13:27.882173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.882118 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752"} err="failed to get container status \"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752\": rpc error: code = NotFound desc = could not find container \"da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752\": container with ID starting with da2e9a0b098f123cfcdc046fe0c1c06293e192bd76d412741384e51958741752 not found: ID does not exist" Oct 31 01:13:27.941354 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.941320 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:13:27.946078 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:27.946050 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-328d61/auth-test-model-328d61-kserve-79d9c7495c-2m9jw"] Oct 31 01:13:28.643689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:28.643656 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:28.656944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:28.656915 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:28.893011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:28.892978 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:13:29.105274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105241 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:13:29.105847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105831 5011 cpu_manager.go:414] "containerMap: removing container" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerName="main" Oct 31 01:13:29.105903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105848 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerName="main" Oct 31 01:13:29.105903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105875 5011 cpu_manager.go:414] "containerMap: removing container" podUID="aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" containerName="revoke-keys" Oct 31 01:13:29.105903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105881 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" containerName="revoke-keys" Oct 31 01:13:29.106000 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105982 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" containerName="revoke-keys" Oct 31 01:13:29.106000 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.105996 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" containerName="main" Oct 31 01:13:29.110709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.110680 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.137688 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.133296 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:13:29.137688 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.134236 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-36af31-openshift-default-dockercfg-vv85m\"" Oct 31 01:13:29.141301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141265 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141362 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141398 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141495 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141569 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141619 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141662 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141709 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.141865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.141829 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg69s\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243369 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg69s\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243447 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243514 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243618 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243540 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243618 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243681 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243723 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243759 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.243801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.243796 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.244077 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.244020 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.244141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.244057 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.244201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.244171 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.244478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.244454 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.244740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.244718 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.246248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.246223 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.246349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.246318 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.270083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.270055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.270189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.270160 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg69s\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.455663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.455580 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:29.683701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.683667 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5a9000-b16f-4cf0-8571-ecad8628a7c4" path="/var/lib/kubelet/pods/df5a9000-b16f-4cf0-8571-ecad8628a7c4/volumes" Oct 31 01:13:29.789990 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:13:29.789959 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74b1d56_7aca_41ab_812f_2dcc158a8134.slice/crio-3a4468e4548001966605a283bc1557e2ec23c69769b7cc0e73c948e50b55e2ad WatchSource:0}: Error finding container 3a4468e4548001966605a283bc1557e2ec23c69769b7cc0e73c948e50b55e2ad: Status 404 returned error can't find the container with id 3a4468e4548001966605a283bc1557e2ec23c69769b7cc0e73c948e50b55e2ad Oct 31 01:13:29.792889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.792852 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:29.792988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.792927 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:29.792988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.792954 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:29.798797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.798775 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:13:29.893255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.891597 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" event={"ID":"b74b1d56-7aca-41ab-812f-2dcc158a8134","Type":"ContainerStarted","Data":"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0"} Oct 31 01:13:29.893255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.891682 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" event={"ID":"b74b1d56-7aca-41ab-812f-2dcc158a8134","Type":"ContainerStarted","Data":"3a4468e4548001966605a283bc1557e2ec23c69769b7cc0e73c948e50b55e2ad"} Oct 31 01:13:29.940241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:29.940174 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podStartSLOduration=0.940160123 podStartE2EDuration="940.160123ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:29.936889697 +0000 UTC m=+3278.385977335" watchObservedRunningTime="2026-10-31 01:13:29.940160123 +0000 UTC m=+3278.389247759" Oct 31 01:13:30.455872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:30.455839 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:30.457281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:30.457255 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.457426 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:30.457309 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:31.455954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:31.455915 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.456488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:31.455992 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:31.970618 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:13:31.970569 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:13:32.456513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.456476 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.456939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.456539 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:32.859540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.859492 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:13:32.865718 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.865691 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.887986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.887952 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:13:32.892299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.892268 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"openshift-service-ca.crt\"" Oct 31 01:13:32.892430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.892297 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"sub-test-model-f2bdf3-kserve-self-signed-certs\"" Oct 31 01:13:32.892430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.892373 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"default-dockercfg-w6mb7\"" Oct 31 01:13:32.895941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.895856 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f2bdf3\"/\"kube-root-ca.crt\"" Oct 31 01:13:32.898032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.897997 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.898331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.898314 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnv59\" (UniqueName: \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.898463 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.898444 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.898560 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.898524 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.898643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.898624 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:32.898720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:32.898668 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.000785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.000695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.000944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.000817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.000944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.000872 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.001029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.001017 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.001162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.001142 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv59\" (UniqueName: \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.001238 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.001192 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.001826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.001799 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.002098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.002075 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.002380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.002354 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.008636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.008607 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.008879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.008855 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.034764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.034050 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv59\" (UniqueName: \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\") pod \"sub-test-model-f2bdf3-kserve-747bb97879-c55v6\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.202285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.201923 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:33.456671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.456574 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:33.457276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.457231 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:33.735456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.735372 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:13:33.975943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.975898 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" event={"ID":"b65fd2cd-2192-4325-8e12-d106402bfe7e","Type":"ContainerStarted","Data":"1f2a7cdfdecf4609a88b14e8b784ed2c29308eaf1749db281963cb035ce938db"} Oct 31 01:13:33.976106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:33.975962 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" event={"ID":"b65fd2cd-2192-4325-8e12-d106402bfe7e","Type":"ContainerStarted","Data":"4e859003568ee780522bb05d36ac5a3af17876afaded54761c0c91ac5bdd0a73"} Oct 31 01:13:34.019626 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:34.019521 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" podStartSLOduration=2.019501986 podStartE2EDuration="2.019501986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:34.016635061 +0000 UTC m=+3282.465722699" watchObservedRunningTime="2026-10-31 01:13:34.019501986 +0000 UTC m=+3282.468589627" Oct 31 01:13:34.202609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:34.202556 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:34.458175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:34.456791 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.458175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:34.456848 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:35.456788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:35.456749 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 31 01:13:35.457013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:35.456822 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 31 01:13:36.461970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:36.461938 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:36.462961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:36.462926 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:36.464068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:36.464043 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:13:43.203165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:43.203130 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:43.247872 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:43.247834 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:44.074671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:44.074642 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:13:50.881723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:50.881679 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:13:52.291842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.291801 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm"] Oct 31 01:13:52.297704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.297673 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.315313 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.315286 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm"] Oct 31 01:13:52.321751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.321728 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:13:52.321898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.321728 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:52.460234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.458890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.460234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.458968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fv9\" (UniqueName: \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.460234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.459234 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.567280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.561391 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.567280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.561557 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.567280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.561632 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fv9\" (UniqueName: \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.567280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.563417 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.571004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.570975 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.590922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.590641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fv9\" (UniqueName: \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\") pod \"maas-api-revoke-sub-b89d45cc-qq7xm\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.612788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.612756 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:52.621666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.621640 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:13:52.638328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:52.638306 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:13:53.047899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:53.047867 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm"] Oct 31 01:13:53.198065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:53.198023 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" event={"ID":"99bb5a7d-53a0-488e-8dfa-e76a57b94c46","Type":"ContainerStarted","Data":"9ffb43a5aa914120ea44587729be61303671fca6e67806bd2f2f35554c8f9613"} Oct 31 01:13:54.207967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.207931 5011 generic.go:356] "Generic (PLEG): container finished" podID="99bb5a7d-53a0-488e-8dfa-e76a57b94c46" containerID="756f01bdc41f7a755bca5ff114effb8843e59bbd05a9d3b2f31cf7dc04e66f08" exitCode=0 Oct 31 01:13:54.208359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.208013 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" event={"ID":"99bb5a7d-53a0-488e-8dfa-e76a57b94c46","Type":"ContainerDied","Data":"756f01bdc41f7a755bca5ff114effb8843e59bbd05a9d3b2f31cf7dc04e66f08"} Oct 31 01:13:54.290099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.290067 5011 scope.go:122] "RemoveContainer" containerID="4fa8e827ce3c64f2fa627db0b73eb908efc7bf60d72096af9cb73400bde2e81f" Oct 31 01:13:54.300655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.300631 5011 scope.go:122] "RemoveContainer" containerID="489b02a268aedce52785e933da103401ffecbfcd9e2b904c31fe7f93a705a623" Oct 31 01:13:54.310095 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.310072 5011 scope.go:122] "RemoveContainer" containerID="41f10a0414251cc6d96812a60040e7dde68b0ee6d24bf347583a5fccd28a5771" Oct 31 01:13:54.346934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:54.346902 5011 scope.go:122] "RemoveContainer" containerID="4f10c263a0c4b4527df8a5b5b753d113c4ec615876815f4c9b953cee9966341a" Oct 31 01:13:55.512906 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.512875 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:55.641232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.639868 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\" (UniqueName: \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\") pod \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " Oct 31 01:13:55.641232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.640045 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\") pod \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " Oct 31 01:13:55.641232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.640151 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\") pod \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\" (UID: \"99bb5a7d-53a0-488e-8dfa-e76a57b94c46\") " Oct 31 01:13:55.645228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.643452 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca" pod "99bb5a7d-53a0-488e-8dfa-e76a57b94c46" (UID: "99bb5a7d-53a0-488e-8dfa-e76a57b94c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:55.646143 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.646110 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca" pod "99bb5a7d-53a0-488e-8dfa-e76a57b94c46" (UID: "99bb5a7d-53a0-488e-8dfa-e76a57b94c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:55.648604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.648578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9" pod "99bb5a7d-53a0-488e-8dfa-e76a57b94c46" (UID: "99bb5a7d-53a0-488e-8dfa-e76a57b94c46"). InnerVolumeSpecName "kube-api-access-49fv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:55.747284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.743416 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49fv9\" (UniqueName: \"kubernetes.io/projected/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-kube-api-access-49fv9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:55.747284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.743454 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:55.747284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:55.743474 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bb5a7d-53a0-488e-8dfa-e76a57b94c46-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:13:56.250317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:56.250280 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" Oct 31 01:13:56.250504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:56.250280 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm" event={"ID":"99bb5a7d-53a0-488e-8dfa-e76a57b94c46","Type":"ContainerDied","Data":"9ffb43a5aa914120ea44587729be61303671fca6e67806bd2f2f35554c8f9613"} Oct 31 01:13:56.250504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:13:56.250408 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ffb43a5aa914120ea44587729be61303671fca6e67806bd2f2f35554c8f9613" Oct 31 01:14:00.817934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:00.817887 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:14:00.818508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:00.818166 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" podUID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" containerName="payload-processing" containerID="cri-o://063d83286ea6ce662e13a14cc5a4108ec399ac7382a570cf434c544f4ee10d8a" gracePeriod=30 Oct 31 01:14:01.315620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.315519 5011 generic.go:356] "Generic (PLEG): container finished" podID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" containerID="063d83286ea6ce662e13a14cc5a4108ec399ac7382a570cf434c544f4ee10d8a" exitCode=0 Oct 31 01:14:01.316051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.316024 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" event={"ID":"c4f7bc64-4532-4757-b8d6-732b4e6b5a01","Type":"ContainerDied","Data":"063d83286ea6ce662e13a14cc5a4108ec399ac7382a570cf434c544f4ee10d8a"} Oct 31 01:14:01.316248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.316228 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" event={"ID":"c4f7bc64-4532-4757-b8d6-732b4e6b5a01","Type":"ContainerDied","Data":"5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca"} Oct 31 01:14:01.317124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.317095 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba3b811116db9b28b84c7c26e95789a492093dfa6961f2e6d9c113347f349ca" Oct 31 01:14:01.330561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.330533 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:14:01.342090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.341107 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\" (UniqueName: \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\") pod \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " Oct 31 01:14:01.342090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.341440 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\") pod \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\" (UID: \"c4f7bc64-4532-4757-b8d6-732b4e6b5a01\") " Oct 31 01:14:01.342871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.342534 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume" pod "c4f7bc64-4532-4757-b8d6-732b4e6b5a01" (UID: "c4f7bc64-4532-4757-b8d6-732b4e6b5a01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:01.349533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.349493 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb" pod "c4f7bc64-4532-4757-b8d6-732b4e6b5a01" (UID: "c4f7bc64-4532-4757-b8d6-732b4e6b5a01"). InnerVolumeSpecName "kube-api-access-6t2mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:01.428216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.428163 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j"] Oct 31 01:14:01.432393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.432364 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-73f915-czr4j"] Oct 31 01:14:01.443535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.443504 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.443535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.443534 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t2mb\" (UniqueName: \"kubernetes.io/projected/c4f7bc64-4532-4757-b8d6-732b4e6b5a01-kube-api-access-6t2mb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.684689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:01.684657 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb" path="/var/lib/kubelet/pods/aee0ac7c-cc4c-4b1d-90f0-f77a966e61cb/volumes" Oct 31 01:14:01.947861 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:14:01.947782 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:14:02.325241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.325194 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx" Oct 31 01:14:02.381706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.381671 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:14:02.391679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.391648 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-6f7644dc56-wchnx"] Oct 31 01:14:02.688593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.688517 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp"] Oct 31 01:14:02.689133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689119 5011 cpu_manager.go:414] "containerMap: removing container" podUID="99bb5a7d-53a0-488e-8dfa-e76a57b94c46" containerName="revoke-keys" Oct 31 01:14:02.689184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689134 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bb5a7d-53a0-488e-8dfa-e76a57b94c46" containerName="revoke-keys" Oct 31 01:14:02.689184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689149 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" containerName="payload-processing" Oct 31 01:14:02.689184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689154 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" containerName="payload-processing" Oct 31 01:14:02.689330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689317 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bb5a7d-53a0-488e-8dfa-e76a57b94c46" containerName="revoke-keys" Oct 31 01:14:02.689369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.689335 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" containerName="payload-processing" Oct 31 01:14:02.692841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.692798 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.707969 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.707947 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:02.708128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.708028 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:14:02.713763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.713743 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp"] Oct 31 01:14:02.763368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.763334 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.763559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.763396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjw5\" (UniqueName: \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.763631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.763611 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.864690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.864660 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.864864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.864715 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjw5\" (UniqueName: \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.864864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.864813 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.865401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.865378 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.867154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.867128 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:02.888629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:02.888602 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjw5\" (UniqueName: \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\") pod \"maas-api-revoke-sub-53f910c4-vf4vp\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:03.007451 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.007365 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:03.335796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.335762 5011 generic.go:356] "Generic (PLEG): container finished" podID="23dba605-8432-495d-a997-50c8b6338fa1" containerID="1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b" exitCode=0 Oct 31 01:14:03.335998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.335832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerDied","Data":"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b"} Oct 31 01:14:03.336415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.336395 5011 scope.go:122] "RemoveContainer" containerID="1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b" Oct 31 01:14:03.339986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.339946 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:03.340126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.340029 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:03.340126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.340066 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:03.340512 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:14:03.340485 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7161847_a1d8_458b_8fbf_fb033b291ac3.slice/crio-4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587 WatchSource:0}: Error finding container 4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587: Status 404 returned error can't find the container with id 4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587 Oct 31 01:14:03.341022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.340998 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp"] Oct 31 01:14:03.684218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:03.684182 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f7bc64-4532-4757-b8d6-732b4e6b5a01" path="/var/lib/kubelet/pods/c4f7bc64-4532-4757-b8d6-732b4e6b5a01/volumes" Oct 31 01:14:04.353096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:04.353061 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerStarted","Data":"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73"} Oct 31 01:14:04.355028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:04.354998 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" event={"ID":"d7161847-a1d8-458b-8fbf-fb033b291ac3","Type":"ContainerStarted","Data":"d23655d1c20e6c285791ebb487844c1e57ec29c2095db7624a5b636de5be7b2f"} Oct 31 01:14:04.355157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:04.355054 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" event={"ID":"d7161847-a1d8-458b-8fbf-fb033b291ac3","Type":"ContainerStarted","Data":"4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587"} Oct 31 01:14:05.271680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:05.271646 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:14:05.271869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:05.271787 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 31 01:14:05.271869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:05.271830 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 31 01:14:05.367904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:05.367865 5011 generic.go:356] "Generic (PLEG): container finished" podID="d7161847-a1d8-458b-8fbf-fb033b291ac3" containerID="d23655d1c20e6c285791ebb487844c1e57ec29c2095db7624a5b636de5be7b2f" exitCode=0 Oct 31 01:14:05.369868 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:05.369834 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" event={"ID":"d7161847-a1d8-458b-8fbf-fb033b291ac3","Type":"ContainerDied","Data":"d23655d1c20e6c285791ebb487844c1e57ec29c2095db7624a5b636de5be7b2f"} Oct 31 01:14:06.271586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.271547 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 31 01:14:06.271784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.271604 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 31 01:14:06.605851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.605826 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:06.765124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.765094 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\" (UniqueName: \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\") pod \"d7161847-a1d8-458b-8fbf-fb033b291ac3\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " Oct 31 01:14:06.765556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.765531 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\") pod \"d7161847-a1d8-458b-8fbf-fb033b291ac3\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " Oct 31 01:14:06.765857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.765836 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\") pod \"d7161847-a1d8-458b-8fbf-fb033b291ac3\" (UID: \"d7161847-a1d8-458b-8fbf-fb033b291ac3\") " Oct 31 01:14:06.766350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.766316 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca" pod "d7161847-a1d8-458b-8fbf-fb033b291ac3" (UID: "d7161847-a1d8-458b-8fbf-fb033b291ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:06.767473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.767432 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:06.768359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.768333 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5" pod "d7161847-a1d8-458b-8fbf-fb033b291ac3" (UID: "d7161847-a1d8-458b-8fbf-fb033b291ac3"). InnerVolumeSpecName "kube-api-access-9cjw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:06.774247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.769597 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca" pod "d7161847-a1d8-458b-8fbf-fb033b291ac3" (UID: "d7161847-a1d8-458b-8fbf-fb033b291ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:06.868921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.868832 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjw5\" (UniqueName: \"kubernetes.io/projected/d7161847-a1d8-458b-8fbf-fb033b291ac3-kube-api-access-9cjw5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:06.868921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:06.868865 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7161847-a1d8-458b-8fbf-fb033b291ac3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:07.271771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:07.271683 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 31 01:14:07.271771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:07.271749 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 31 01:14:07.387338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:07.387305 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" event={"ID":"d7161847-a1d8-458b-8fbf-fb033b291ac3","Type":"ContainerDied","Data":"4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587"} Oct 31 01:14:07.387521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:07.387337 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp" Oct 31 01:14:07.387521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:07.387355 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb00cca47ad3cc3de710003b71d0d012d8b47456652b14c36ac202f80391587" Oct 31 01:14:08.272407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:08.272367 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 31 01:14:08.272915 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:08.272436 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 31 01:14:09.272233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:09.271592 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 31 01:14:09.272233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:09.271671 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 31 01:14:10.242421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:10.242388 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:14:10.277826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:10.277785 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:14:10.428735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:10.428704 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:14:13.663505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:13.663466 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:14:14.985776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:14.985733 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z"] Oct 31 01:14:14.986533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:14.986510 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d7161847-a1d8-458b-8fbf-fb033b291ac3" containerName="revoke-keys" Oct 31 01:14:14.986533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:14.986531 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7161847-a1d8-458b-8fbf-fb033b291ac3" containerName="revoke-keys" Oct 31 01:14:14.986713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:14.986672 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7161847-a1d8-458b-8fbf-fb033b291ac3" containerName="revoke-keys" Oct 31 01:14:14.991534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:14.991508 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.007968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.007945 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z"] Oct 31 01:14:15.014615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.014592 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:15.014851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.014833 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:14:15.111362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.111321 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvl4\" (UniqueName: \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.111571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.111512 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.111571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.111556 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.212635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.212597 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvl4\" (UniqueName: \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.212806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.212697 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.212806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.212724 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.213418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.213383 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.215139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.215116 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.236333 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.236276 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvl4\" (UniqueName: \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\") pod \"maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.309225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.309193 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:15.629755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:15.629718 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z"] Oct 31 01:14:15.630036 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:14:15.630001 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fd193c_0ec7_4d45_b8f8_e0686e7e8147.slice/crio-ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51 WatchSource:0}: Error finding container ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51: Status 404 returned error can't find the container with id ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51 Oct 31 01:14:16.479258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:16.479180 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" event={"ID":"92fd193c-0ec7-4d45-b8f8-e0686e7e8147","Type":"ContainerStarted","Data":"c45f9ed6b1e34b509a1e75ccc529623565010af249d1ab5c88ccf8bd6568eff9"} Oct 31 01:14:16.479706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:16.479285 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" event={"ID":"92fd193c-0ec7-4d45-b8f8-e0686e7e8147","Type":"ContainerStarted","Data":"ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51"} Oct 31 01:14:16.523035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:16.522986 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" podStartSLOduration=2.5229732719999998 podStartE2EDuration="2.522973272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:16.521560299 +0000 UTC m=+3324.970647940" watchObservedRunningTime="2026-10-31 01:14:16.522973272 +0000 UTC m=+3324.972060910" Oct 31 01:14:17.500033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:17.499998 5011 generic.go:356] "Generic (PLEG): container finished" podID="92fd193c-0ec7-4d45-b8f8-e0686e7e8147" containerID="c45f9ed6b1e34b509a1e75ccc529623565010af249d1ab5c88ccf8bd6568eff9" exitCode=0 Oct 31 01:14:17.500480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:17.500088 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" event={"ID":"92fd193c-0ec7-4d45-b8f8-e0686e7e8147","Type":"ContainerDied","Data":"c45f9ed6b1e34b509a1e75ccc529623565010af249d1ab5c88ccf8bd6568eff9"} Oct 31 01:14:18.689847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.689824 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:18.785246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.785151 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\" (UniqueName: \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\") pod \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " Oct 31 01:14:18.785406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.785263 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\") pod \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " Oct 31 01:14:18.785406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.785368 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\") pod \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\" (UID: \"92fd193c-0ec7-4d45-b8f8-e0686e7e8147\") " Oct 31 01:14:18.785655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.785627 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca" pod "92fd193c-0ec7-4d45-b8f8-e0686e7e8147" (UID: "92fd193c-0ec7-4d45-b8f8-e0686e7e8147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:18.786174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.786148 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:18.787786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.787760 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4" pod "92fd193c-0ec7-4d45-b8f8-e0686e7e8147" (UID: "92fd193c-0ec7-4d45-b8f8-e0686e7e8147"). InnerVolumeSpecName "kube-api-access-nbvl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:18.787878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.787793 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca" pod "92fd193c-0ec7-4d45-b8f8-e0686e7e8147" (UID: "92fd193c-0ec7-4d45-b8f8-e0686e7e8147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:18.887625 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.887595 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:18.887625 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:18.887625 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbvl4\" (UniqueName: \"kubernetes.io/projected/92fd193c-0ec7-4d45-b8f8-e0686e7e8147-kube-api-access-nbvl4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:19.516379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:19.516343 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" Oct 31 01:14:19.516379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:19.516362 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z" event={"ID":"92fd193c-0ec7-4d45-b8f8-e0686e7e8147","Type":"ContainerDied","Data":"ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51"} Oct 31 01:14:19.516609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:19.516420 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7223d9b8a8b72057dbfc4dd15570e64a6bb73954f3e208016ddcc4562deb51" Oct 31 01:14:27.428565 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.426537 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:14:27.504174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.504116 5011 cpu_manager.go:414] "containerMap: removing container" podUID="92fd193c-0ec7-4d45-b8f8-e0686e7e8147" containerName="revoke-keys" Oct 31 01:14:27.504457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.504434 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fd193c-0ec7-4d45-b8f8-e0686e7e8147" containerName="revoke-keys" Oct 31 01:14:27.504851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.504822 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fd193c-0ec7-4d45-b8f8-e0686e7e8147" containerName="revoke-keys" Oct 31 01:14:27.576454 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.576421 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:14:27.576676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.576571 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.603889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.603863 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fa97ea-openshift-default-dockercfg-pktbz\"" Oct 31 01:14:27.658389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658351 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658409 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658648 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658761 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658714 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99w8\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658783 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658821 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.658867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.658849 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762304 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762386 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762443 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762498 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99w8\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762564 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762620 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.763281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.762663 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.764397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.764366 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.765910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.764643 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.770319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.764976 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.770446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.765132 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.770546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.765850 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.780237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.778619 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.780237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.779522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.787265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.786616 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.787904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.787883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99w8\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\") pod \"e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:27.894595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:27.894550 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:28.252369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.252328 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:14:28.253783 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:14:28.253748 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a09d8a_7353_4172_a8a4_c7f76c5e6dc3.slice/crio-ec14f1a1ce3159321f22974226a1a4dc9752bb456fbab647ad5cd47dfc63ba2f WatchSource:0}: Error finding container ec14f1a1ce3159321f22974226a1a4dc9752bb456fbab647ad5cd47dfc63ba2f: Status 404 returned error can't find the container with id ec14f1a1ce3159321f22974226a1a4dc9752bb456fbab647ad5cd47dfc63ba2f Oct 31 01:14:28.257609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.257570 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:28.257702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.257659 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:28.257741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.257700 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:28.609274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.609221 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerStarted","Data":"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1"} Oct 31 01:14:28.609662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.609289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerStarted","Data":"ec14f1a1ce3159321f22974226a1a4dc9752bb456fbab647ad5cd47dfc63ba2f"} Oct 31 01:14:28.682606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.682536 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podStartSLOduration=1.68251996 podStartE2EDuration="1.68251996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:28.670330292 +0000 UTC m=+3337.119417928" watchObservedRunningTime="2026-10-31 01:14:28.68251996 +0000 UTC m=+3337.131607600" Oct 31 01:14:28.687533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.687386 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerName="main" containerID="cri-o://d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293" gracePeriod=45 Oct 31 01:14:28.895368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:28.895294 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:29.176936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.176906 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:14:29.188482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188452 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.188609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188515 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.188609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188586 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.188685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188668 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\" (UniqueName: \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.188789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188774 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.188865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.188823 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\") pod \"e09d287e-3aad-464b-ba4b-11252ed5b03f\" (UID: \"e09d287e-3aad-464b-ba4b-11252ed5b03f\") " Oct 31 01:14:29.189430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.189408 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:29.189946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.189924 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:29.190084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.190065 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:29.191824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.191802 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:29.192267 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.192243 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:29.192603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.192587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss" pod "e09d287e-3aad-464b-ba4b-11252ed5b03f" (UID: "e09d287e-3aad-464b-ba4b-11252ed5b03f"). InnerVolumeSpecName "kube-api-access-k4kss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:29.291338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291300 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09d287e-3aad-464b-ba4b-11252ed5b03f-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.291338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291330 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.291338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291339 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.291570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291347 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kss\" (UniqueName: \"kubernetes.io/projected/e09d287e-3aad-464b-ba4b-11252ed5b03f-kube-api-access-k4kss\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.291570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291357 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.291570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.291366 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09d287e-3aad-464b-ba4b-11252ed5b03f-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:14:29.628235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.624949 5011 generic.go:356] "Generic (PLEG): container finished" podID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerID="d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293" exitCode=0 Oct 31 01:14:29.628235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.626235 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" Oct 31 01:14:29.628235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.626254 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" event={"ID":"e09d287e-3aad-464b-ba4b-11252ed5b03f","Type":"ContainerDied","Data":"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293"} Oct 31 01:14:29.628235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.626307 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" event={"ID":"e09d287e-3aad-464b-ba4b-11252ed5b03f","Type":"ContainerDied","Data":"e68d8b6d56b0de5ef087b0e1b17fb79b1b4c619e4385ebd11f6346da523dd5dc"} Oct 31 01:14:29.628235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.626333 5011 scope.go:122] "RemoveContainer" containerID="d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293" Oct 31 01:14:29.656595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.651030 5011 scope.go:122] "RemoveContainer" containerID="d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293" Oct 31 01:14:29.656595 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:14:29.651430 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293\": container with ID starting with d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293 not found: ID does not exist" containerID="d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293" Oct 31 01:14:29.656595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.651468 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293"} err="failed to get container status \"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293\": rpc error: code = NotFound desc = could not find container \"d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293\": container with ID starting with d8b3e2acb824a453992f2ba1f8919387189861bb5eaf505d7a1af4415dafe293 not found: ID does not exist" Oct 31 01:14:29.702321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.702283 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:14:29.712481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.712445 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv"] Oct 31 01:14:29.901605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:29.901530 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:30.036230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:30.033298 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-bcdb4e/rate-test-model-bcdb4e-kserve-7d6b8bdf65-wrnnv" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.228:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:30.655099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:30.655060 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:30.656634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:30.656605 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:14:31.683984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:31.683951 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" path="/var/lib/kubelet/pods/e09d287e-3aad-464b-ba4b-11252ed5b03f/volumes" Oct 31 01:14:31.973706 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:14:31.973603 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:14:33.072147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.072106 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:14:33.072794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.072776 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerName="main" Oct 31 01:14:33.072794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.072794 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerName="main" Oct 31 01:14:33.072924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.072902 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09d287e-3aad-464b-ba4b-11252ed5b03f" containerName="main" Oct 31 01:14:33.077193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.077167 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.099714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.099685 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:14:33.112963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.112919 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa97ea\"" Oct 31 01:14:33.113130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.113106 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa97ea-dockercfg-xqpzr\"" Oct 31 01:14:33.225280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.189775 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fgd\" (UniqueName: \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.232237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.226970 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.340735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.337313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fgd\" (UniqueName: \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.340735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.337367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.340735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.338470 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.363113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.362993 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fgd\" (UniqueName: \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\") pod \"payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.392314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.392284 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:33.715353 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:14:33.715323 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b15e37d_d3af_4e86_978b_a2f1fbc87a61.slice/crio-2956ac822f4a6727a37a299118457fcfbba24a43965883eead1af49d0ea532f7 WatchSource:0}: Error finding container 2956ac822f4a6727a37a299118457fcfbba24a43965883eead1af49d0ea532f7: Status 404 returned error can't find the container with id 2956ac822f4a6727a37a299118457fcfbba24a43965883eead1af49d0ea532f7 Oct 31 01:14:33.717940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:33.717908 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:14:34.690276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:34.690185 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" event={"ID":"7b15e37d-d3af-4e86-978b-a2f1fbc87a61","Type":"ContainerStarted","Data":"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed"} Oct 31 01:14:34.690784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:34.690299 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" event={"ID":"7b15e37d-d3af-4e86-978b-a2f1fbc87a61","Type":"ContainerStarted","Data":"2956ac822f4a6727a37a299118457fcfbba24a43965883eead1af49d0ea532f7"} Oct 31 01:14:34.690784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:34.690426 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:34.734081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:34.734012 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" podStartSLOduration=1.733996339 podStartE2EDuration="1.733996339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:34.732749945 +0000 UTC m=+3343.181837584" watchObservedRunningTime="2026-10-31 01:14:34.733996339 +0000 UTC m=+3343.183083980" Oct 31 01:14:37.750079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.750037 5011 generic.go:356] "Generic (PLEG): container finished" podID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerID="112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740" exitCode=0 Oct 31 01:14:37.750605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.750107 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerDied","Data":"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740"} Oct 31 01:14:37.750726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.750709 5011 scope.go:122] "RemoveContainer" containerID="112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740" Oct 31 01:14:37.753081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.753049 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:37.753219 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.753140 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:37.753219 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:37.753184 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:38.761852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:38.761799 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerStarted","Data":"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c"} Oct 31 01:14:38.845622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:38.845587 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:14:38.846832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:38.845914 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:38.847048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:38.847024 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:39.847229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:39.846333 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:39.847229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:39.846397 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:40.848012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:40.847549 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:40.848012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:40.847612 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:41.850240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:41.844729 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:41.850240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:41.844804 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:42.847598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:42.847556 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:42.847900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:42.847870 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:43.850572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:43.850526 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 01:14:43.852115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:43.852077 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 01:14:44.849821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:44.849787 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:14:44.850323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:44.850308 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:14:44.851141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:44.851111 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:14:45.713908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:45.713880 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:14:54.400439 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:14:54.400403 5011 scope.go:122] "RemoveContainer" containerID="c7e4dccda2c51fb0475628a189781aa7766a146bc668829843c83ce02ca2f13d" Oct 31 01:15:00.149775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.149740 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:00.155557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.155532 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:00.155687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.155665 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.161648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.161622 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:00.161784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.161720 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.166578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.166542 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:00.166695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.166665 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.171361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.171342 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:00.171534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.171475 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.176269 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.176246 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.190005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.189982 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:15:00.193024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.192940 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:00.193294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.193271 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.193679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.193561 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpcz\" (UniqueName: \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.199136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.199114 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:00.213229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.209629 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.220262 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.216322 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:00.222380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.221313 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:00.226292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.225242 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:00.229899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.229868 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:00.237645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.236466 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:00.252400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.252367 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:00.255756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.253119 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.257887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.257863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:00.258000 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.257965 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.285890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.285860 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:00.290537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.290356 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295570 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295603 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbhk\" (UniqueName: \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r4m\" (UniqueName: \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295699 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295746 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2p8\" (UniqueName: \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295763 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52t5q\" (UniqueName: \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295788 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295832 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gzr\" (UniqueName: \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295877 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295908 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfl9x\" (UniqueName: \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295923 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnpb\" (UniqueName: \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295944 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.295988 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpcz\" (UniqueName: \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.296011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.296029 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.297274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.296710 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.300833 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.300808 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:00.301304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.301278 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.306039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.306006 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:00.306243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.306198 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.326963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.326936 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:00.339413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.332515 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:00.339413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.332522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpcz\" (UniqueName: \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.339413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.336857 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:00.347799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.347769 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:00.348585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.348560 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.367874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.365495 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:00.367874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.365887 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.372266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.371267 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:00.372266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.371354 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.378480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.378424 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:00.384388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.384129 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:00.396871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.396842 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rs2d\" (UniqueName: \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.396978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.396891 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r4m\" (UniqueName: \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.397036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.396978 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.397096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397030 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbc2t\" (UniqueName: \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.397158 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2p8\" (UniqueName: \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.397158 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397139 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52t5q\" (UniqueName: \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.397317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397177 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.397544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397518 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gzr\" (UniqueName: \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.397643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397571 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.397707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397661 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfl9x\" (UniqueName: \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.397707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397690 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnpb\" (UniqueName: \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.397815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397760 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.397864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397841 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.397938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397909 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.398020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397973 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.398020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.397980 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.398020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398005 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.398200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398097 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.398200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398164 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.398363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398201 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.398363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398288 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbhk\" (UniqueName: \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.398363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398342 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6d95\" (UniqueName: \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.398661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.398739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398686 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.398739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.398698 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.399253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.399189 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.399253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.399191 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.424636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.424513 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r4m\" (UniqueName: \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.424636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.424534 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2p8\" (UniqueName: \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.424636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.424542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnpb\" (UniqueName: \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.424636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.424586 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52t5q\" (UniqueName: \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.425007 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.424991 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gzr\" (UniqueName: \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.426024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.425997 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfl9x\" (UniqueName: \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.426243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.426226 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbhk\" (UniqueName: \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.472842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.472819 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:00.482074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.482051 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:00.499766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.499739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.499884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.499814 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zflk\" (UniqueName: \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.499884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.499878 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.500017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.499906 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.500017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500005 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.500136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500040 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.500199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500133 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6d95\" (UniqueName: \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.500199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rs2d\" (UniqueName: \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.500394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500251 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbc2t\" (UniqueName: \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.500394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500281 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6qd\" (UniqueName: \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.500548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.500523 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.501188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.501164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.501382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.501302 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.518381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.518353 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:00.524332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.524302 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:00.528248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.526612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rs2d\" (UniqueName: \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.528248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.527480 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6d95\" (UniqueName: \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\") pod \"maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.560842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.547487 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbc2t\" (UniqueName: \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.560842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.547704 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:00.560842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.560372 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:00.575925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.575460 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:00.601886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.601846 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zflk\" (UniqueName: \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.602005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.601938 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.602072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.602051 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.602228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.602193 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6qd\" (UniqueName: \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.603075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.603048 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.603322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.603298 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.610462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.610436 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:00.634201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.634168 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zflk\" (UniqueName: \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\") pod \"maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.634453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.634432 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:00.635706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.635665 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6qd\" (UniqueName: \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\") pod \"maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.662744 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.662481 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:00.677374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.677041 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:00.695815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.695537 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:00.716508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.704531 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:00.871620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.843892 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:00.959052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:00.959017 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerStarted","Data":"7a2bc4cab93472459e2e677feeb32e25397cc7580b8b90e97cd035313d460872"} Oct 31 01:15:01.044989 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.044958 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65df237a_12c1_431a_b296_c6511639ac6a.slice/crio-fbf6d474748b5fa1ed0622e868a17c2d4533aa9f8d9ac59aa331725f44845149 WatchSource:0}: Error finding container fbf6d474748b5fa1ed0622e868a17c2d4533aa9f8d9ac59aa331725f44845149: Status 404 returned error can't find the container with id fbf6d474748b5fa1ed0622e868a17c2d4533aa9f8d9ac59aa331725f44845149 Oct 31 01:15:01.045365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.045337 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:01.176548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.176517 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:01.234421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.234386 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:01.243567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.243543 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:01.248541 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.247867 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170cd7e9_031e_491e_8b97_c71acf4703e5.slice/crio-3b4561d0515f79fffc6f178f833da10c67ee8749c7d30fdbd523441afa8c3789 WatchSource:0}: Error finding container 3b4561d0515f79fffc6f178f833da10c67ee8749c7d30fdbd523441afa8c3789: Status 404 returned error can't find the container with id 3b4561d0515f79fffc6f178f833da10c67ee8749c7d30fdbd523441afa8c3789 Oct 31 01:15:01.248541 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.248131 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64aa2b3a_3cf4_4833_9158_ee0669837f86.slice/crio-17a413f95863636b4eaf66067d4c6c066b770b13da741da9fc83cf8d47a989e4 WatchSource:0}: Error finding container 17a413f95863636b4eaf66067d4c6c066b770b13da741da9fc83cf8d47a989e4: Status 404 returned error can't find the container with id 17a413f95863636b4eaf66067d4c6c066b770b13da741da9fc83cf8d47a989e4 Oct 31 01:15:01.250372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.250277 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:01.254477 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.254435 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:01.299604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.299577 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:01.302704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.302686 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:01.308053 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.308020 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8356a45b_2097_440f_b6ae_698d125ab90a.slice/crio-ef4f0e4af695cf8e6d805a8e6af074c595bcef446fc3806e81a7da5318f86535 WatchSource:0}: Error finding container ef4f0e4af695cf8e6d805a8e6af074c595bcef446fc3806e81a7da5318f86535: Status 404 returned error can't find the container with id ef4f0e4af695cf8e6d805a8e6af074c595bcef446fc3806e81a7da5318f86535 Oct 31 01:15:01.345369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.345345 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:01.365721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.365691 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:01.432084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.432057 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:01.446501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.446478 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:01.522877 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.522809 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01eeafe6_ab6d_452d_ab2e_aefce14b5a22.slice/crio-6a16e830355499d24cc11b2264335626c0640a3dba11ae80b577781b9d6edf40 WatchSource:0}: Error finding container 6a16e830355499d24cc11b2264335626c0640a3dba11ae80b577781b9d6edf40: Status 404 returned error can't find the container with id 6a16e830355499d24cc11b2264335626c0640a3dba11ae80b577781b9d6edf40 Oct 31 01:15:01.529500 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.529469 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26caeb57_fce5_4ac5_adb9_ce621d9f2be4.slice/crio-afc3d8254cfdaa6d2816ca70e4f2c5ec00678024b396e2ce6d3b259a29c2080d WatchSource:0}: Error finding container afc3d8254cfdaa6d2816ca70e4f2c5ec00678024b396e2ce6d3b259a29c2080d: Status 404 returned error can't find the container with id afc3d8254cfdaa6d2816ca70e4f2c5ec00678024b396e2ce6d3b259a29c2080d Oct 31 01:15:01.531254 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:01.530802 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a08fcb_2bf0_4856_ae94_672abb421881.slice/crio-1f88494742a0494999dbd5995cd3940adcaf514cd3e0cc4316581c5b6f592432 WatchSource:0}: Error finding container 1f88494742a0494999dbd5995cd3940adcaf514cd3e0cc4316581c5b6f592432: Status 404 returned error can't find the container with id 1f88494742a0494999dbd5995cd3940adcaf514cd3e0cc4316581c5b6f592432 Oct 31 01:15:01.969362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.969331 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerStarted","Data":"66826ea3c7b40ed2f6c50dd7e368b104f1ab04312e263fc4a9a54c161f91bd9f"} Oct 31 01:15:01.970756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.970728 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerStarted","Data":"3903493d46015d7e0e3a8fd932d866d788024fb3eb3a1dc45cab2500e5c7e377"} Oct 31 01:15:01.972187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.972159 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerStarted","Data":"ccebe9b6f401aab45e58912aa09cbf86aa3e94a1c585121be463afe4f043bd5e"} Oct 31 01:15:01.973513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.973487 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerStarted","Data":"d884744a8170281831d987df2702830210ce668ac0ee9e7e52db3c91e8a1b509"} Oct 31 01:15:01.975087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.975065 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerStarted","Data":"4ad955d7d77701e2c15477f7c65f60481d4efcb818958ee05507e3abbf84e352"} Oct 31 01:15:01.976811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.976785 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerStarted","Data":"69556743811c8bfe90139b8a4fe5ffc90b7d3b44e107f0f1bd4b62224cbf6f13"} Oct 31 01:15:01.976910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.976837 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerStarted","Data":"3b4561d0515f79fffc6f178f833da10c67ee8749c7d30fdbd523441afa8c3789"} Oct 31 01:15:01.978458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.978433 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerStarted","Data":"0ea996ee7203c17ae4c387b13adc80237629cebff5af103fd7bf0a37379b57ed"} Oct 31 01:15:01.978576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.978478 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerStarted","Data":"fbf6d474748b5fa1ed0622e868a17c2d4533aa9f8d9ac59aa331725f44845149"} Oct 31 01:15:01.979734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.979710 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerStarted","Data":"1f88494742a0494999dbd5995cd3940adcaf514cd3e0cc4316581c5b6f592432"} Oct 31 01:15:01.980935 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.980912 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerStarted","Data":"6a16e830355499d24cc11b2264335626c0640a3dba11ae80b577781b9d6edf40"} Oct 31 01:15:01.982162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.982140 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerStarted","Data":"40f14331bb43ad6cc4062ee5a3e88ac68d63f8b55a387ea47f188a5f4104dea6"} Oct 31 01:15:01.983457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.983424 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerStarted","Data":"ef4f0e4af695cf8e6d805a8e6af074c595bcef446fc3806e81a7da5318f86535"} Oct 31 01:15:01.985241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.985186 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerStarted","Data":"204cd9a61ca0a35412666ea8b551f838a979ea90eaf3f1ea86107b03143bfd24"} Oct 31 01:15:01.985364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.985265 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerStarted","Data":"17a413f95863636b4eaf66067d4c6c066b770b13da741da9fc83cf8d47a989e4"} Oct 31 01:15:01.986651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:01.986630 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerStarted","Data":"afc3d8254cfdaa6d2816ca70e4f2c5ec00678024b396e2ce6d3b259a29c2080d"} Oct 31 01:15:02.027257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.027176 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" podStartSLOduration=2.027163113 podStartE2EDuration="2.027163113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.023533917 +0000 UTC m=+3370.472621557" watchObservedRunningTime="2026-10-31 01:15:02.027163113 +0000 UTC m=+3370.476250751" Oct 31 01:15:02.075913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.075864 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" podStartSLOduration=2.075850589 podStartE2EDuration="2.075850589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.074787776 +0000 UTC m=+3370.523875415" watchObservedRunningTime="2026-10-31 01:15:02.075850589 +0000 UTC m=+3370.524938235" Oct 31 01:15:02.125690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.125574 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" podStartSLOduration=2.1255580370000002 podStartE2EDuration="2.125558037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.123176369 +0000 UTC m=+3370.572264008" watchObservedRunningTime="2026-10-31 01:15:02.125558037 +0000 UTC m=+3370.574645677" Oct 31 01:15:02.167883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.167819 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" podStartSLOduration=2.167798903 podStartE2EDuration="2.167798903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.165670226 +0000 UTC m=+3370.614757866" watchObservedRunningTime="2026-10-31 01:15:02.167798903 +0000 UTC m=+3370.616886542" Oct 31 01:15:02.224326 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:02.224122 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:15:02.996614 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.996581 5011 generic.go:356] "Generic (PLEG): container finished" podID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerID="204cd9a61ca0a35412666ea8b551f838a979ea90eaf3f1ea86107b03143bfd24" exitCode=60 Oct 31 01:15:02.996800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.996665 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerDied","Data":"204cd9a61ca0a35412666ea8b551f838a979ea90eaf3f1ea86107b03143bfd24"} Oct 31 01:15:02.996974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.996952 5011 scope.go:122] "RemoveContainer" containerID="204cd9a61ca0a35412666ea8b551f838a979ea90eaf3f1ea86107b03143bfd24" Oct 31 01:15:02.998646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.998624 5011 generic.go:356] "Generic (PLEG): container finished" podID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerID="660ded46b178e6ca79aed0f38a25dceb091ee6439ea871e811849504e4d071e1" exitCode=60 Oct 31 01:15:02.998764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.998708 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerDied","Data":"660ded46b178e6ca79aed0f38a25dceb091ee6439ea871e811849504e4d071e1"} Oct 31 01:15:02.998948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:02.998934 5011 scope.go:122] "RemoveContainer" containerID="660ded46b178e6ca79aed0f38a25dceb091ee6439ea871e811849504e4d071e1" Oct 31 01:15:03.001227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.001189 5011 generic.go:356] "Generic (PLEG): container finished" podID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerID="49834e0e1afc6cd06ec3faa961faa873abb97ced6c7ddcbd83bc4b733948eef2" exitCode=60 Oct 31 01:15:03.001311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.001295 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerDied","Data":"49834e0e1afc6cd06ec3faa961faa873abb97ced6c7ddcbd83bc4b733948eef2"} Oct 31 01:15:03.001635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.001618 5011 scope.go:122] "RemoveContainer" containerID="49834e0e1afc6cd06ec3faa961faa873abb97ced6c7ddcbd83bc4b733948eef2" Oct 31 01:15:03.003687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.003666 5011 generic.go:356] "Generic (PLEG): container finished" podID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerID="0bb8adc661e438c6da6fc3fde5c6ff416dbd13ccdb82586c29d63bf3b7b48cd3" exitCode=60 Oct 31 01:15:03.003786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.003749 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerDied","Data":"0bb8adc661e438c6da6fc3fde5c6ff416dbd13ccdb82586c29d63bf3b7b48cd3"} Oct 31 01:15:03.003956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.003937 5011 scope.go:122] "RemoveContainer" containerID="0bb8adc661e438c6da6fc3fde5c6ff416dbd13ccdb82586c29d63bf3b7b48cd3" Oct 31 01:15:03.005564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.005542 5011 generic.go:356] "Generic (PLEG): container finished" podID="149338af-dbda-433a-b141-dd78426fde67" containerID="bebf0b97f6bde1fc3d8e08cdbd8c2c3d82940c82950b734e665717663aba38d2" exitCode=60 Oct 31 01:15:03.005642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.005624 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerDied","Data":"bebf0b97f6bde1fc3d8e08cdbd8c2c3d82940c82950b734e665717663aba38d2"} Oct 31 01:15:03.005827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.005811 5011 scope.go:122] "RemoveContainer" containerID="bebf0b97f6bde1fc3d8e08cdbd8c2c3d82940c82950b734e665717663aba38d2" Oct 31 01:15:03.007359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.007337 5011 generic.go:356] "Generic (PLEG): container finished" podID="0202194e-2baf-429f-a03b-e50d04d3c286" containerID="e75c47c65bf614f57a5f736af9c290d92956d766671b9ebe32c0177bb2a95960" exitCode=60 Oct 31 01:15:03.007447 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.007419 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerDied","Data":"e75c47c65bf614f57a5f736af9c290d92956d766671b9ebe32c0177bb2a95960"} Oct 31 01:15:03.007605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.007588 5011 scope.go:122] "RemoveContainer" containerID="e75c47c65bf614f57a5f736af9c290d92956d766671b9ebe32c0177bb2a95960" Oct 31 01:15:03.009194 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.009158 5011 generic.go:356] "Generic (PLEG): container finished" podID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerID="4ad955d7d77701e2c15477f7c65f60481d4efcb818958ee05507e3abbf84e352" exitCode=60 Oct 31 01:15:03.009300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.009263 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerDied","Data":"4ad955d7d77701e2c15477f7c65f60481d4efcb818958ee05507e3abbf84e352"} Oct 31 01:15:03.009462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.009442 5011 scope.go:122] "RemoveContainer" containerID="4ad955d7d77701e2c15477f7c65f60481d4efcb818958ee05507e3abbf84e352" Oct 31 01:15:03.011944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.011921 5011 generic.go:356] "Generic (PLEG): container finished" podID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerID="69556743811c8bfe90139b8a4fe5ffc90b7d3b44e107f0f1bd4b62224cbf6f13" exitCode=60 Oct 31 01:15:03.012054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.011999 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerDied","Data":"69556743811c8bfe90139b8a4fe5ffc90b7d3b44e107f0f1bd4b62224cbf6f13"} Oct 31 01:15:03.012440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.012198 5011 scope.go:122] "RemoveContainer" containerID="69556743811c8bfe90139b8a4fe5ffc90b7d3b44e107f0f1bd4b62224cbf6f13" Oct 31 01:15:03.014057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.014037 5011 generic.go:356] "Generic (PLEG): container finished" podID="65df237a-12c1-431a-b296-c6511639ac6a" containerID="0ea996ee7203c17ae4c387b13adc80237629cebff5af103fd7bf0a37379b57ed" exitCode=60 Oct 31 01:15:03.014159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.014107 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerDied","Data":"0ea996ee7203c17ae4c387b13adc80237629cebff5af103fd7bf0a37379b57ed"} Oct 31 01:15:03.014366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.014350 5011 scope.go:122] "RemoveContainer" containerID="0ea996ee7203c17ae4c387b13adc80237629cebff5af103fd7bf0a37379b57ed" Oct 31 01:15:03.015773 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.015753 5011 generic.go:356] "Generic (PLEG): container finished" podID="27a08fcb-2bf0-4856-ae94-672abb421881" containerID="3eff994f0eca6e3d34da29694c3ac2976f94588ea70fb97c2f946e1eeb50d1d1" exitCode=60 Oct 31 01:15:03.015886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.015833 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerDied","Data":"3eff994f0eca6e3d34da29694c3ac2976f94588ea70fb97c2f946e1eeb50d1d1"} Oct 31 01:15:03.016013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.016000 5011 scope.go:122] "RemoveContainer" containerID="3eff994f0eca6e3d34da29694c3ac2976f94588ea70fb97c2f946e1eeb50d1d1" Oct 31 01:15:03.017424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.017400 5011 generic.go:356] "Generic (PLEG): container finished" podID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerID="fc503b3c9f635bb3bbc32569eae633a7f153045d130fe63978d895a8afb1b822" exitCode=60 Oct 31 01:15:03.017521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.017488 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerDied","Data":"fc503b3c9f635bb3bbc32569eae633a7f153045d130fe63978d895a8afb1b822"} Oct 31 01:15:03.017660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.017646 5011 scope.go:122] "RemoveContainer" containerID="fc503b3c9f635bb3bbc32569eae633a7f153045d130fe63978d895a8afb1b822" Oct 31 01:15:03.019245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.019201 5011 generic.go:356] "Generic (PLEG): container finished" podID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerID="dc6f682ab7a9ac3beaa9a3b99cd5491bf80c44d234c1695487995fcfc4c7f128" exitCode=60 Oct 31 01:15:03.019508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.019451 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerDied","Data":"dc6f682ab7a9ac3beaa9a3b99cd5491bf80c44d234c1695487995fcfc4c7f128"} Oct 31 01:15:03.019613 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.019528 5011 scope.go:122] "RemoveContainer" containerID="dc6f682ab7a9ac3beaa9a3b99cd5491bf80c44d234c1695487995fcfc4c7f128" Oct 31 01:15:03.031107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.031084 5011 generic.go:356] "Generic (PLEG): container finished" podID="8356a45b-2097-440f-b6ae-698d125ab90a" containerID="92617c12b09f56b3d3b24ae58f7ed4d7c7cc3880af84572278bdf90b65323af5" exitCode=60 Oct 31 01:15:03.031178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.031146 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerDied","Data":"92617c12b09f56b3d3b24ae58f7ed4d7c7cc3880af84572278bdf90b65323af5"} Oct 31 01:15:03.031588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:03.031568 5011 scope.go:122] "RemoveContainer" containerID="92617c12b09f56b3d3b24ae58f7ed4d7c7cc3880af84572278bdf90b65323af5" Oct 31 01:15:04.054327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.053776 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerStarted","Data":"f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573"} Oct 31 01:15:04.059794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.058922 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerStarted","Data":"12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d"} Oct 31 01:15:04.062360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.062307 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerStarted","Data":"6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06"} Oct 31 01:15:04.065732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.065255 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerStarted","Data":"9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4"} Oct 31 01:15:04.069040 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.068748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerStarted","Data":"a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c"} Oct 31 01:15:04.071026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.070968 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerStarted","Data":"e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529"} Oct 31 01:15:04.073982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.073882 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerStarted","Data":"89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a"} Oct 31 01:15:04.081889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.081418 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerStarted","Data":"a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258"} Oct 31 01:15:04.083682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.083660 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerStarted","Data":"1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed"} Oct 31 01:15:04.086786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.086278 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerStarted","Data":"05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115"} Oct 31 01:15:04.090480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.090459 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerStarted","Data":"ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846"} Oct 31 01:15:04.141228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.139379 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerStarted","Data":"b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539"} Oct 31 01:15:04.160325 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.160295 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:15:04.160521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.160498 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" podUID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" containerName="payload-processing" containerID="cri-o://b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03" gracePeriod=30 Oct 31 01:15:04.202497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.202434 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" podStartSLOduration=4.202413787 podStartE2EDuration="4.202413787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.192697508 +0000 UTC m=+3372.641785148" watchObservedRunningTime="2026-10-31 01:15:04.202413787 +0000 UTC m=+3372.651501427" Oct 31 01:15:04.290369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.289174 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" podStartSLOduration=4.289152986 podStartE2EDuration="4.289152986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.238595611 +0000 UTC m=+3372.687683253" watchObservedRunningTime="2026-10-31 01:15:04.289152986 +0000 UTC m=+3372.738240626" Oct 31 01:15:04.361254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.357015 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" podStartSLOduration=4.35699868 podStartE2EDuration="4.35699868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.33677249 +0000 UTC m=+3372.785860131" watchObservedRunningTime="2026-10-31 01:15:04.35699868 +0000 UTC m=+3372.806086329" Oct 31 01:15:04.435696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.435618 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" podStartSLOduration=4.435599172 podStartE2EDuration="4.435599172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.427856117 +0000 UTC m=+3372.876943757" watchObservedRunningTime="2026-10-31 01:15:04.435599172 +0000 UTC m=+3372.884686813" Oct 31 01:15:04.514452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.497784 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" podStartSLOduration=4.497764205 podStartE2EDuration="4.497764205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.479455545 +0000 UTC m=+3372.928543185" watchObservedRunningTime="2026-10-31 01:15:04.497764205 +0000 UTC m=+3372.946851846" Oct 31 01:15:04.561909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.533622 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" podStartSLOduration=4.533601356 podStartE2EDuration="4.533601356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.527646202 +0000 UTC m=+3372.976733842" watchObservedRunningTime="2026-10-31 01:15:04.533601356 +0000 UTC m=+3372.982688997" Oct 31 01:15:04.764601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.703026 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" podStartSLOduration=4.703011385 podStartE2EDuration="4.703011385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.697969499 +0000 UTC m=+3373.147057149" watchObservedRunningTime="2026-10-31 01:15:04.703011385 +0000 UTC m=+3373.152099051" Oct 31 01:15:04.805551 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.805502 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" podStartSLOduration=4.805483624 podStartE2EDuration="4.805483624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.804374471 +0000 UTC m=+3373.253462110" watchObservedRunningTime="2026-10-31 01:15:04.805483624 +0000 UTC m=+3373.254571263" Oct 31 01:15:04.928314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.928271 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z"] Oct 31 01:15:04.932314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:04.932291 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75ead8-pgv4z"] Oct 31 01:15:05.149956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.149889 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:15:05.155052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.155018 5011 generic.go:356] "Generic (PLEG): container finished" podID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerID="a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c" exitCode=60 Oct 31 01:15:05.155197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.155107 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerDied","Data":"a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c"} Oct 31 01:15:05.155197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.155185 5011 scope.go:122] "RemoveContainer" containerID="fc503b3c9f635bb3bbc32569eae633a7f153045d130fe63978d895a8afb1b822" Oct 31 01:15:05.155402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.155383 5011 scope.go:122] "RemoveContainer" containerID="a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c" Oct 31 01:15:05.155804 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.155762 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr_odh-ai-gateway-infra(01eeafe6-ab6d-452d-ab2e-aefce14b5a22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" Oct 31 01:15:05.157635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.157611 5011 generic.go:356] "Generic (PLEG): container finished" podID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerID="e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529" exitCode=60 Oct 31 01:15:05.157754 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.157742 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerDied","Data":"e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529"} Oct 31 01:15:05.158046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.158026 5011 scope.go:122] "RemoveContainer" containerID="e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529" Oct 31 01:15:05.158397 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.158376 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt_odh-ai-gateway-infra(ebdf7ea4-71eb-4f66-9311-c4af74dabb3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" Oct 31 01:15:05.159988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.159964 5011 generic.go:356] "Generic (PLEG): container finished" podID="8356a45b-2097-440f-b6ae-698d125ab90a" containerID="89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a" exitCode=60 Oct 31 01:15:05.160291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.160268 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerDied","Data":"89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a"} Oct 31 01:15:05.160481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.160465 5011 scope.go:122] "RemoveContainer" containerID="89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a" Oct 31 01:15:05.161862 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.160817 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck_odh-ai-gateway-infra(8356a45b-2097-440f-b6ae-698d125ab90a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" Oct 31 01:15:05.165604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.165581 5011 generic.go:356] "Generic (PLEG): container finished" podID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerID="a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258" exitCode=60 Oct 31 01:15:05.165735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.165668 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerDied","Data":"a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258"} Oct 31 01:15:05.165904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.165887 5011 scope.go:122] "RemoveContainer" containerID="a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258" Oct 31 01:15:05.166273 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.166189 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp_odh-ai-gateway-infra(64aa2b3a-3cf4-4833-9158-ee0669837f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" Oct 31 01:15:05.167959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.167934 5011 generic.go:356] "Generic (PLEG): container finished" podID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerID="1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed" exitCode=60 Oct 31 01:15:05.168074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.168020 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerDied","Data":"1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed"} Oct 31 01:15:05.168302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.168285 5011 scope.go:122] "RemoveContainer" containerID="1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed" Oct 31 01:15:05.168597 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.168578 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp_odh-ai-gateway-infra(26caeb57-fce5-4ac5-adb9-ce621d9f2be4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" Oct 31 01:15:05.169512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.169488 5011 generic.go:356] "Generic (PLEG): container finished" podID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" containerID="b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03" exitCode=0 Oct 31 01:15:05.169607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.169531 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" Oct 31 01:15:05.169607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.169565 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" event={"ID":"020d2d90-4a77-4a24-a4ac-52b4ae6f1439","Type":"ContainerDied","Data":"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03"} Oct 31 01:15:05.169607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.169603 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv" event={"ID":"020d2d90-4a77-4a24-a4ac-52b4ae6f1439","Type":"ContainerDied","Data":"b7cb5d931c136f740ef3c5b77edcc96ab2e324a7bedad2fb6b01342f170aaf05"} Oct 31 01:15:05.171873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.171787 5011 generic.go:356] "Generic (PLEG): container finished" podID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerID="5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f" exitCode=60 Oct 31 01:15:05.171873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.171866 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerDied","Data":"5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f"} Oct 31 01:15:05.172133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.172119 5011 scope.go:122] "RemoveContainer" containerID="5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f" Oct 31 01:15:05.172468 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.172445 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh_odh-ai-gateway-infra(0cbe5cf5-78ac-47de-b445-42e717b05f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" Oct 31 01:15:05.173058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.173041 5011 scope.go:122] "RemoveContainer" containerID="dc6f682ab7a9ac3beaa9a3b99cd5491bf80c44d234c1695487995fcfc4c7f128" Oct 31 01:15:05.174134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.174114 5011 generic.go:356] "Generic (PLEG): container finished" podID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerID="05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115" exitCode=60 Oct 31 01:15:05.174224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.174182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerDied","Data":"05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115"} Oct 31 01:15:05.174395 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.174378 5011 scope.go:122] "RemoveContainer" containerID="05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115" Oct 31 01:15:05.174630 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.174608 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66_odh-ai-gateway-infra(64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" Oct 31 01:15:05.176151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.176130 5011 generic.go:356] "Generic (PLEG): container finished" podID="149338af-dbda-433a-b141-dd78426fde67" containerID="ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846" exitCode=60 Oct 31 01:15:05.176264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.176222 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerDied","Data":"ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846"} Oct 31 01:15:05.176445 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.176426 5011 scope.go:122] "RemoveContainer" containerID="ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846" Oct 31 01:15:05.176698 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.176675 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx_odh-ai-gateway-infra(149338af-dbda-433a-b141-dd78426fde67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" podUID="149338af-dbda-433a-b141-dd78426fde67" Oct 31 01:15:05.178122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.178104 5011 generic.go:356] "Generic (PLEG): container finished" podID="0202194e-2baf-429f-a03b-e50d04d3c286" containerID="b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539" exitCode=60 Oct 31 01:15:05.178217 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.178178 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerDied","Data":"b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539"} Oct 31 01:15:05.178545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.178526 5011 scope.go:122] "RemoveContainer" containerID="b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539" Oct 31 01:15:05.178806 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.178784 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h_odh-ai-gateway-infra(0202194e-2baf-429f-a03b-e50d04d3c286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" Oct 31 01:15:05.180104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.180080 5011 generic.go:356] "Generic (PLEG): container finished" podID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerID="f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573" exitCode=60 Oct 31 01:15:05.180199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.180158 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerDied","Data":"f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573"} Oct 31 01:15:05.180373 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.180356 5011 scope.go:122] "RemoveContainer" containerID="f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573" Oct 31 01:15:05.180615 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.180589 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp_odh-ai-gateway-infra(a9efcb13-2067-4675-8a03-d131feaeefe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" Oct 31 01:15:05.182415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.182394 5011 generic.go:356] "Generic (PLEG): container finished" podID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerID="12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d" exitCode=60 Oct 31 01:15:05.182556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.182481 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerDied","Data":"12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d"} Oct 31 01:15:05.183365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.183343 5011 scope.go:122] "RemoveContainer" containerID="12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d" Oct 31 01:15:05.183589 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.183570 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl_odh-ai-gateway-infra(170cd7e9-031e-491e-8b97-c71acf4703e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" Oct 31 01:15:05.184517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.184496 5011 generic.go:356] "Generic (PLEG): container finished" podID="65df237a-12c1-431a-b296-c6511639ac6a" containerID="6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06" exitCode=60 Oct 31 01:15:05.184595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.184581 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerDied","Data":"6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06"} Oct 31 01:15:05.184748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.184733 5011 scope.go:122] "RemoveContainer" containerID="6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06" Oct 31 01:15:05.184945 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.184920 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn_odh-ai-gateway-infra(65df237a-12c1-431a-b296-c6511639ac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" podUID="65df237a-12c1-431a-b296-c6511639ac6a" Oct 31 01:15:05.186693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.186671 5011 generic.go:356] "Generic (PLEG): container finished" podID="27a08fcb-2bf0-4856-ae94-672abb421881" containerID="9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4" exitCode=60 Oct 31 01:15:05.186783 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.186711 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerDied","Data":"9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4"} Oct 31 01:15:05.186960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.186945 5011 scope.go:122] "RemoveContainer" containerID="9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4" Oct 31 01:15:05.187200 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.187181 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw_odh-ai-gateway-infra(27a08fcb-2bf0-4856-ae94-672abb421881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" Oct 31 01:15:05.189440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.189420 5011 scope.go:122] "RemoveContainer" containerID="92617c12b09f56b3d3b24ae58f7ed4d7c7cc3880af84572278bdf90b65323af5" Oct 31 01:15:05.206338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.206317 5011 scope.go:122] "RemoveContainer" containerID="204cd9a61ca0a35412666ea8b551f838a979ea90eaf3f1ea86107b03143bfd24" Oct 31 01:15:05.221015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.220993 5011 scope.go:122] "RemoveContainer" containerID="660ded46b178e6ca79aed0f38a25dceb091ee6439ea871e811849504e4d071e1" Oct 31 01:15:05.237821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.237789 5011 scope.go:122] "RemoveContainer" containerID="b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03" Oct 31 01:15:05.244078 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.244051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\" (UniqueName: \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\") pod \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " Oct 31 01:15:05.244290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.244268 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\") pod \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\" (UID: \"020d2d90-4a77-4a24-a4ac-52b4ae6f1439\") " Oct 31 01:15:05.244864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.244820 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume" pod "020d2d90-4a77-4a24-a4ac-52b4ae6f1439" (UID: "020d2d90-4a77-4a24-a4ac-52b4ae6f1439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:05.246629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.246611 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:05.247071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.247050 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg" pod "020d2d90-4a77-4a24-a4ac-52b4ae6f1439" (UID: "020d2d90-4a77-4a24-a4ac-52b4ae6f1439"). InnerVolumeSpecName "kube-api-access-x8swg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:05.252965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.252940 5011 scope.go:122] "RemoveContainer" containerID="b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03" Oct 31 01:15:05.253279 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:05.253256 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03\": container with ID starting with b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03 not found: ID does not exist" containerID="b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03" Oct 31 01:15:05.253379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.253290 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03"} err="failed to get container status \"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03\": rpc error: code = NotFound desc = could not find container \"b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03\": container with ID starting with b16141912e3d17b2b9112b8ea124ad0df83773cdef496d490f2a655483320f03 not found: ID does not exist" Oct 31 01:15:05.253379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.253314 5011 scope.go:122] "RemoveContainer" containerID="49834e0e1afc6cd06ec3faa961faa873abb97ced6c7ddcbd83bc4b733948eef2" Oct 31 01:15:05.270051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.270025 5011 scope.go:122] "RemoveContainer" containerID="0bb8adc661e438c6da6fc3fde5c6ff416dbd13ccdb82586c29d63bf3b7b48cd3" Oct 31 01:15:05.286850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.286827 5011 scope.go:122] "RemoveContainer" containerID="bebf0b97f6bde1fc3d8e08cdbd8c2c3d82940c82950b734e665717663aba38d2" Oct 31 01:15:05.301501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.301480 5011 scope.go:122] "RemoveContainer" containerID="e75c47c65bf614f57a5f736af9c290d92956d766671b9ebe32c0177bb2a95960" Oct 31 01:15:05.315039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.315013 5011 scope.go:122] "RemoveContainer" containerID="4ad955d7d77701e2c15477f7c65f60481d4efcb818958ee05507e3abbf84e352" Oct 31 01:15:05.329531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.329510 5011 scope.go:122] "RemoveContainer" containerID="69556743811c8bfe90139b8a4fe5ffc90b7d3b44e107f0f1bd4b62224cbf6f13" Oct 31 01:15:05.343957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.343935 5011 scope.go:122] "RemoveContainer" containerID="0ea996ee7203c17ae4c387b13adc80237629cebff5af103fd7bf0a37379b57ed" Oct 31 01:15:05.347808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.347781 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8swg\" (UniqueName: \"kubernetes.io/projected/020d2d90-4a77-4a24-a4ac-52b4ae6f1439-kube-api-access-x8swg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:05.358854 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.358833 5011 scope.go:122] "RemoveContainer" containerID="3eff994f0eca6e3d34da29694c3ac2976f94588ea70fb97c2f946e1eeb50d1d1" Oct 31 01:15:05.684841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.684808 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fd193c-0ec7-4d45-b8f8-e0686e7e8147" path="/var/lib/kubelet/pods/92fd193c-0ec7-4d45-b8f8-e0686e7e8147/volumes" Oct 31 01:15:05.826498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.826459 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:15:05.833135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:05.833089 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75ead8-7c5767987b-gp4gv"] Oct 31 01:15:06.196996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.196910 5011 scope.go:122] "RemoveContainer" containerID="b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539" Oct 31 01:15:06.197417 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.197281 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h_odh-ai-gateway-infra(0202194e-2baf-429f-a03b-e50d04d3c286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" Oct 31 01:15:06.198684 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.198663 5011 scope.go:122] "RemoveContainer" containerID="f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573" Oct 31 01:15:06.198957 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.198936 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp_odh-ai-gateway-infra(a9efcb13-2067-4675-8a03-d131feaeefe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" Oct 31 01:15:06.200301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.200282 5011 scope.go:122] "RemoveContainer" containerID="12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d" Oct 31 01:15:06.200583 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.200562 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl_odh-ai-gateway-infra(170cd7e9-031e-491e-8b97-c71acf4703e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" Oct 31 01:15:06.201856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.201836 5011 scope.go:122] "RemoveContainer" containerID="6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06" Oct 31 01:15:06.202112 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.202084 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn_odh-ai-gateway-infra(65df237a-12c1-431a-b296-c6511639ac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" podUID="65df237a-12c1-431a-b296-c6511639ac6a" Oct 31 01:15:06.203345 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.203329 5011 scope.go:122] "RemoveContainer" containerID="9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4" Oct 31 01:15:06.203579 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.203558 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw_odh-ai-gateway-infra(27a08fcb-2bf0-4856-ae94-672abb421881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" Oct 31 01:15:06.204972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.204950 5011 scope.go:122] "RemoveContainer" containerID="a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c" Oct 31 01:15:06.205257 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.205201 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr_odh-ai-gateway-infra(01eeafe6-ab6d-452d-ab2e-aefce14b5a22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" Oct 31 01:15:06.206483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.206464 5011 scope.go:122] "RemoveContainer" containerID="e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529" Oct 31 01:15:06.206702 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.206680 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt_odh-ai-gateway-infra(ebdf7ea4-71eb-4f66-9311-c4af74dabb3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" Oct 31 01:15:06.208146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.208129 5011 scope.go:122] "RemoveContainer" containerID="89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a" Oct 31 01:15:06.208408 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.208376 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck_odh-ai-gateway-infra(8356a45b-2097-440f-b6ae-698d125ab90a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" Oct 31 01:15:06.209923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.209904 5011 scope.go:122] "RemoveContainer" containerID="a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258" Oct 31 01:15:06.210155 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.210135 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp_odh-ai-gateway-infra(64aa2b3a-3cf4-4833-9158-ee0669837f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" Oct 31 01:15:06.211536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.211517 5011 scope.go:122] "RemoveContainer" containerID="1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed" Oct 31 01:15:06.211742 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.211717 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp_odh-ai-gateway-infra(26caeb57-fce5-4ac5-adb9-ce621d9f2be4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" Oct 31 01:15:06.214265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.214249 5011 scope.go:122] "RemoveContainer" containerID="5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f" Oct 31 01:15:06.214452 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.214433 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh_odh-ai-gateway-infra(0cbe5cf5-78ac-47de-b445-42e717b05f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" Oct 31 01:15:06.216180 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.216160 5011 scope.go:122] "RemoveContainer" containerID="05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115" Oct 31 01:15:06.216420 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.216397 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66_odh-ai-gateway-infra(64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" Oct 31 01:15:06.217750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:06.217728 5011 scope.go:122] "RemoveContainer" containerID="ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846" Oct 31 01:15:06.217962 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:06.217935 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx_odh-ai-gateway-infra(149338af-dbda-433a-b141-dd78426fde67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" podUID="149338af-dbda-433a-b141-dd78426fde67" Oct 31 01:15:07.684371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:07.684337 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" path="/var/lib/kubelet/pods/020d2d90-4a77-4a24-a4ac-52b4ae6f1439/volumes" Oct 31 01:15:15.676276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.676241 5011 scope.go:122] "RemoveContainer" containerID="f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573" Oct 31 01:15:15.676742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.676405 5011 scope.go:122] "RemoveContainer" containerID="e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529" Oct 31 01:15:15.676742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.676724 5011 scope.go:122] "RemoveContainer" containerID="a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c" Oct 31 01:15:15.676925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.676906 5011 scope.go:122] "RemoveContainer" containerID="05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115" Oct 31 01:15:15.677379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.677016 5011 scope.go:122] "RemoveContainer" containerID="9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4" Oct 31 01:15:15.677379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.677133 5011 scope.go:122] "RemoveContainer" containerID="5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f" Oct 31 01:15:15.677493 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:15.677461 5011 scope.go:122] "RemoveContainer" containerID="6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06" Oct 31 01:15:16.393138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.393104 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerStarted","Data":"ba70a63af5fc8f6e06c2ad1e15de983933d6e3ed20c775ce257c1eab20e7311e"} Oct 31 01:15:16.395335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.395308 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerStarted","Data":"ddc3c4124342bb8fd01801f21a207c7e7711a0e0bf9831a7dfe817c9140fb410"} Oct 31 01:15:16.397245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.397220 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerStarted","Data":"cd0dacd140d8c02f63e05be3928ab8e20f0fd15016adca37f84aa28dc5d56f2b"} Oct 31 01:15:16.399102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.399082 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerStarted","Data":"f04d6ad09de7e241d3751a821324ef66a4742fce926acb629804b935be94ed87"} Oct 31 01:15:16.401199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.401174 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerStarted","Data":"254a410b5b6cacc35100d03e936a260fcf656b9cd3b94dfd3b9e6753fbc21c75"} Oct 31 01:15:16.408560 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.408535 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerStarted","Data":"ca11d5f0b4c44ba864b5bb99cc02ade8517243a9aaa92a8386cd63aa7c2269cf"} Oct 31 01:15:16.412073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.412053 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerStarted","Data":"ef9f1c199321aa238cf57fbc62a01caed021c3da66c1431e13b29860a95c18c4"} Oct 31 01:15:16.545734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.545634 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" podStartSLOduration=16.545620402 podStartE2EDuration="16.545620402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:16.542045641 +0000 UTC m=+3384.991133280" watchObservedRunningTime="2026-10-31 01:15:16.545620402 +0000 UTC m=+3384.994708041" Oct 31 01:15:16.676407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:16.676376 5011 scope.go:122] "RemoveContainer" containerID="a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258" Oct 31 01:15:17.425729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.425693 5011 generic.go:356] "Generic (PLEG): container finished" podID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerID="ca11d5f0b4c44ba864b5bb99cc02ade8517243a9aaa92a8386cd63aa7c2269cf" exitCode=60 Oct 31 01:15:17.425893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.425754 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerDied","Data":"ca11d5f0b4c44ba864b5bb99cc02ade8517243a9aaa92a8386cd63aa7c2269cf"} Oct 31 01:15:17.425893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.425837 5011 scope.go:122] "RemoveContainer" containerID="5a297a4598d362c7664868c213a504b3aa77fed8f5802d24c73b7adaf229cb1f" Oct 31 01:15:17.426332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.426308 5011 scope.go:122] "RemoveContainer" containerID="ca11d5f0b4c44ba864b5bb99cc02ade8517243a9aaa92a8386cd63aa7c2269cf" Oct 31 01:15:17.426822 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.426706 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh_odh-ai-gateway-infra(0cbe5cf5-78ac-47de-b445-42e717b05f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" Oct 31 01:15:17.428353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.428326 5011 generic.go:356] "Generic (PLEG): container finished" podID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerID="ef9f1c199321aa238cf57fbc62a01caed021c3da66c1431e13b29860a95c18c4" exitCode=60 Oct 31 01:15:17.428460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.428408 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerDied","Data":"ef9f1c199321aa238cf57fbc62a01caed021c3da66c1431e13b29860a95c18c4"} Oct 31 01:15:17.428853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.428831 5011 scope.go:122] "RemoveContainer" containerID="ef9f1c199321aa238cf57fbc62a01caed021c3da66c1431e13b29860a95c18c4" Oct 31 01:15:17.429247 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.429202 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66_odh-ai-gateway-infra(64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" Oct 31 01:15:17.431350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.431322 5011 generic.go:356] "Generic (PLEG): container finished" podID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerID="ba70a63af5fc8f6e06c2ad1e15de983933d6e3ed20c775ce257c1eab20e7311e" exitCode=60 Oct 31 01:15:17.431469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.431384 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerDied","Data":"ba70a63af5fc8f6e06c2ad1e15de983933d6e3ed20c775ce257c1eab20e7311e"} Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.431829 5011 scope.go:122] "RemoveContainer" containerID="ba70a63af5fc8f6e06c2ad1e15de983933d6e3ed20c775ce257c1eab20e7311e" Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.432183 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp_odh-ai-gateway-infra(a9efcb13-2067-4675-8a03-d131feaeefe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.434075 5011 generic.go:356] "Generic (PLEG): container finished" podID="65df237a-12c1-431a-b296-c6511639ac6a" containerID="ddc3c4124342bb8fd01801f21a207c7e7711a0e0bf9831a7dfe817c9140fb410" exitCode=60 Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.434143 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerDied","Data":"ddc3c4124342bb8fd01801f21a207c7e7711a0e0bf9831a7dfe817c9140fb410"} Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.434537 5011 scope.go:122] "RemoveContainer" containerID="ddc3c4124342bb8fd01801f21a207c7e7711a0e0bf9831a7dfe817c9140fb410" Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.434869 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn_odh-ai-gateway-infra(65df237a-12c1-431a-b296-c6511639ac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" podUID="65df237a-12c1-431a-b296-c6511639ac6a" Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.437185 5011 generic.go:356] "Generic (PLEG): container finished" podID="27a08fcb-2bf0-4856-ae94-672abb421881" containerID="cd0dacd140d8c02f63e05be3928ab8e20f0fd15016adca37f84aa28dc5d56f2b" exitCode=60 Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.437263 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerDied","Data":"cd0dacd140d8c02f63e05be3928ab8e20f0fd15016adca37f84aa28dc5d56f2b"} Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.437630 5011 scope.go:122] "RemoveContainer" containerID="cd0dacd140d8c02f63e05be3928ab8e20f0fd15016adca37f84aa28dc5d56f2b" Oct 31 01:15:17.438221 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.437932 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw_odh-ai-gateway-infra(27a08fcb-2bf0-4856-ae94-672abb421881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" Oct 31 01:15:17.440532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.440511 5011 generic.go:356] "Generic (PLEG): container finished" podID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerID="f04d6ad09de7e241d3751a821324ef66a4742fce926acb629804b935be94ed87" exitCode=60 Oct 31 01:15:17.440628 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.440574 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerDied","Data":"f04d6ad09de7e241d3751a821324ef66a4742fce926acb629804b935be94ed87"} Oct 31 01:15:17.440870 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.440857 5011 scope.go:122] "RemoveContainer" containerID="f04d6ad09de7e241d3751a821324ef66a4742fce926acb629804b935be94ed87" Oct 31 01:15:17.441151 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.441123 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr_odh-ai-gateway-infra(01eeafe6-ab6d-452d-ab2e-aefce14b5a22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" Oct 31 01:15:17.443046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.443026 5011 generic.go:356] "Generic (PLEG): container finished" podID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerID="254a410b5b6cacc35100d03e936a260fcf656b9cd3b94dfd3b9e6753fbc21c75" exitCode=60 Oct 31 01:15:17.443141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.443092 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerDied","Data":"254a410b5b6cacc35100d03e936a260fcf656b9cd3b94dfd3b9e6753fbc21c75"} Oct 31 01:15:17.443400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.443385 5011 scope.go:122] "RemoveContainer" containerID="254a410b5b6cacc35100d03e936a260fcf656b9cd3b94dfd3b9e6753fbc21c75" Oct 31 01:15:17.443603 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:17.443583 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt_odh-ai-gateway-infra(ebdf7ea4-71eb-4f66-9311-c4af74dabb3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" Oct 31 01:15:17.444011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.443992 5011 scope.go:122] "RemoveContainer" containerID="05d7856fab9c3713ba50f53fa616b557f0d8ff8c4291dccdafebb27b239da115" Oct 31 01:15:17.458612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.458592 5011 scope.go:122] "RemoveContainer" containerID="f310cb700bc902f5a1ddeecc27757075af24ad72f9ce6e10dc64cac3accb4573" Oct 31 01:15:17.471190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.471167 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:17.473783 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.473765 5011 scope.go:122] "RemoveContainer" containerID="6a890c63e3f2d962ff7012e682ba86bedc78651e07fa8c79178c03cff3e6de06" Oct 31 01:15:17.489557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.489536 5011 scope.go:122] "RemoveContainer" containerID="9df55350e01570206bb1b14193f700a5f1c03631978768f55767e14cbe7529c4" Oct 31 01:15:17.503499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.503482 5011 scope.go:122] "RemoveContainer" containerID="a1d6b6e3fb702632499ec462d9b554f4b434ffb5f9bddc7455ca8ae053528d5c" Oct 31 01:15:17.513968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.513945 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:17.518076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.518057 5011 scope.go:122] "RemoveContainer" containerID="e682fa4d2c0808d71adb85e543cda19a397a5c3ea1c7b0fad4950baa8e3c2529" Oct 31 01:15:17.574063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.574037 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:17.614599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.614524 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:17.684775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.684748 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:17.685598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.685574 5011 scope.go:122] "RemoveContainer" containerID="b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539" Oct 31 01:15:17.686253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.685828 5011 scope.go:122] "RemoveContainer" containerID="89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a" Oct 31 01:15:17.725852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.725824 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:17.756157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:17.756129 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:18.455076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.455034 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerStarted","Data":"546ae4be92c032af4049feb44bc1e3f1066678b7c7c046bab030182d9d700ea7"} Oct 31 01:15:18.457221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.457174 5011 generic.go:356] "Generic (PLEG): container finished" podID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerID="21eb1964e7aa86c583b756b9dce101456a8af5cab08837ee4e9c7e9ce12a6942" exitCode=60 Oct 31 01:15:18.457361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.457252 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerDied","Data":"21eb1964e7aa86c583b756b9dce101456a8af5cab08837ee4e9c7e9ce12a6942"} Oct 31 01:15:18.457361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.457315 5011 scope.go:122] "RemoveContainer" containerID="a4b044c571f70c3ac2376a3a67f5e54055ad35ff73e0c2b103ae6fccee3bd258" Oct 31 01:15:18.457691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.457675 5011 scope.go:122] "RemoveContainer" containerID="21eb1964e7aa86c583b756b9dce101456a8af5cab08837ee4e9c7e9ce12a6942" Oct 31 01:15:18.458038 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:18.458010 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp_odh-ai-gateway-infra(64aa2b3a-3cf4-4833-9158-ee0669837f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" Oct 31 01:15:18.464128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.464103 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerStarted","Data":"8200af195526b7cc03e0f06c46ef98cd2db04b31886521bdd38f1c2f4fc66c05"} Oct 31 01:15:18.659273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.641463 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 01:15:18.659273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.652275 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 01:15:18.675421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.675388 5011 scope.go:122] "RemoveContainer" containerID="12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d" Oct 31 01:15:18.676939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.676190 5011 scope.go:122] "RemoveContainer" containerID="1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed" Oct 31 01:15:18.676939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.676754 5011 scope.go:122] "RemoveContainer" containerID="ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846" Oct 31 01:15:18.961827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.961797 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:18.966608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.966511 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:18.973736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.973678 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:18.977297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.977267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:18.982007 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.981986 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:18.987363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.987337 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:18.992111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:18.992088 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:19.085072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085036 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\" (UniqueName: \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\") pod \"0cbe5cf5-78ac-47de-b445-42e717b05f73\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " Oct 31 01:15:19.085283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085097 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\" (UniqueName: \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\") pod \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " Oct 31 01:15:19.085283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085125 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\") pod \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " Oct 31 01:15:19.085283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085174 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\") pod \"65df237a-12c1-431a-b296-c6511639ac6a\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " Oct 31 01:15:19.085283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085246 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\") pod \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " Oct 31 01:15:19.085536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085311 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\" (UniqueName: \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\") pod \"a9efcb13-2067-4675-8a03-d131feaeefe2\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " Oct 31 01:15:19.085536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085336 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\") pod \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\" (UID: \"01eeafe6-ab6d-452d-ab2e-aefce14b5a22\") " Oct 31 01:15:19.085536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085454 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\") pod \"a9efcb13-2067-4675-8a03-d131feaeefe2\" (UID: \"a9efcb13-2067-4675-8a03-d131feaeefe2\") " Oct 31 01:15:19.085536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085483 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\" (UniqueName: \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\") pod \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\" (UID: \"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a\") " Oct 31 01:15:19.085536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\") pod \"0cbe5cf5-78ac-47de-b445-42e717b05f73\" (UID: \"0cbe5cf5-78ac-47de-b445-42e717b05f73\") " Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085562 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\" (UniqueName: \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\") pod \"27a08fcb-2bf0-4856-ae94-672abb421881\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca" pod "65df237a-12c1-431a-b296-c6511639ac6a" (UID: "65df237a-12c1-431a-b296-c6511639ac6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085599 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca" pod "64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" (UID: "64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085652 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\" (UniqueName: \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\") pod \"65df237a-12c1-431a-b296-c6511639ac6a\" (UID: \"65df237a-12c1-431a-b296-c6511639ac6a\") " Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085756 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\" (UniqueName: \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\") pod \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\" (UID: \"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4\") " Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085788 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\") pod \"27a08fcb-2bf0-4856-ae94-672abb421881\" (UID: \"27a08fcb-2bf0-4856-ae94-672abb421881\") " Oct 31 01:15:19.085806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085797 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca" pod "ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" (UID: "ebdf7ea4-71eb-4f66-9311-c4af74dabb3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.086168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.085883 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca" pod "a9efcb13-2067-4675-8a03-d131feaeefe2" (UID: "a9efcb13-2067-4675-8a03-d131feaeefe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.086168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086157 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca" pod "0cbe5cf5-78ac-47de-b445-42e717b05f73" (UID: "0cbe5cf5-78ac-47de-b445-42e717b05f73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.086290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086231 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca" pod "01eeafe6-ab6d-452d-ab2e-aefce14b5a22" (UID: "01eeafe6-ab6d-452d-ab2e-aefce14b5a22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.086549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086520 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca" pod "27a08fcb-2bf0-4856-ae94-672abb421881" (UID: "27a08fcb-2bf0-4856-ae94-672abb421881"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.086813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086774 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9efcb13-2067-4675-8a03-d131feaeefe2-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086803 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe5cf5-78ac-47de-b445-42e717b05f73-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086817 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a08fcb-2bf0-4856-ae94-672abb421881-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086831 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086846 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df237a-12c1-431a-b296-c6511639ac6a-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086860 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.086997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.086873 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.127290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.122791 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95" pod "ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" (UID: "ebdf7ea4-71eb-4f66-9311-c4af74dabb3a"). InnerVolumeSpecName "kube-api-access-d6d95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.127290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.122874 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr" pod "65df237a-12c1-431a-b296-c6511639ac6a" (UID: "65df237a-12c1-431a-b296-c6511639ac6a"). InnerVolumeSpecName "kube-api-access-r4gzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.127290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.126907 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz" pod "a9efcb13-2067-4675-8a03-d131feaeefe2" (UID: "a9efcb13-2067-4675-8a03-d131feaeefe2"). InnerVolumeSpecName "kube-api-access-qgpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.127290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.126970 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t" pod "64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" (UID: "64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4"). InnerVolumeSpecName "kube-api-access-jbc2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.127290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.127040 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb" pod "0cbe5cf5-78ac-47de-b445-42e717b05f73" (UID: "0cbe5cf5-78ac-47de-b445-42e717b05f73"). InnerVolumeSpecName "kube-api-access-7xnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.140696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.140215 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk" pod "27a08fcb-2bf0-4856-ae94-672abb421881" (UID: "27a08fcb-2bf0-4856-ae94-672abb421881"). InnerVolumeSpecName "kube-api-access-4zflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.140696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.140260 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8" pod "01eeafe6-ab6d-452d-ab2e-aefce14b5a22" (UID: "01eeafe6-ab6d-452d-ab2e-aefce14b5a22"). InnerVolumeSpecName "kube-api-access-zg2p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188711 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbc2t\" (UniqueName: \"kubernetes.io/projected/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4-kube-api-access-jbc2t\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188745 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnpb\" (UniqueName: \"kubernetes.io/projected/0cbe5cf5-78ac-47de-b445-42e717b05f73-kube-api-access-7xnpb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188755 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2p8\" (UniqueName: \"kubernetes.io/projected/01eeafe6-ab6d-452d-ab2e-aefce14b5a22-kube-api-access-zg2p8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188763 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpcz\" (UniqueName: \"kubernetes.io/projected/a9efcb13-2067-4675-8a03-d131feaeefe2-kube-api-access-qgpcz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188773 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6d95\" (UniqueName: \"kubernetes.io/projected/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a-kube-api-access-d6d95\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188782 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zflk\" (UniqueName: \"kubernetes.io/projected/27a08fcb-2bf0-4856-ae94-672abb421881-kube-api-access-4zflk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.192407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.188790 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4gzr\" (UniqueName: \"kubernetes.io/projected/65df237a-12c1-431a-b296-c6511639ac6a-kube-api-access-r4gzr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.487540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.487451 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" Oct 31 01:15:19.487712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.487450 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr" event={"ID":"01eeafe6-ab6d-452d-ab2e-aefce14b5a22","Type":"ContainerDied","Data":"6a16e830355499d24cc11b2264335626c0640a3dba11ae80b577781b9d6edf40"} Oct 31 01:15:19.487712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.487609 5011 scope.go:122] "RemoveContainer" containerID="f04d6ad09de7e241d3751a821324ef66a4742fce926acb629804b935be94ed87" Oct 31 01:15:19.489490 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.489466 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" Oct 31 01:15:19.489635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.489505 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt" event={"ID":"ebdf7ea4-71eb-4f66-9311-c4af74dabb3a","Type":"ContainerDied","Data":"40f14331bb43ad6cc4062ee5a3e88ac68d63f8b55a387ea47f188a5f4104dea6"} Oct 31 01:15:19.491945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.491917 5011 generic.go:356] "Generic (PLEG): container finished" podID="8356a45b-2097-440f-b6ae-698d125ab90a" containerID="546ae4be92c032af4049feb44bc1e3f1066678b7c7c046bab030182d9d700ea7" exitCode=60 Oct 31 01:15:19.492076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.492040 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerDied","Data":"546ae4be92c032af4049feb44bc1e3f1066678b7c7c046bab030182d9d700ea7"} Oct 31 01:15:19.492557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.492533 5011 scope.go:122] "RemoveContainer" containerID="546ae4be92c032af4049feb44bc1e3f1066678b7c7c046bab030182d9d700ea7" Oct 31 01:15:19.492953 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:19.492851 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck_odh-ai-gateway-infra(8356a45b-2097-440f-b6ae-698d125ab90a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" Oct 31 01:15:19.502546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.502531 5011 scope.go:122] "RemoveContainer" containerID="254a410b5b6cacc35100d03e936a260fcf656b9cd3b94dfd3b9e6753fbc21c75" Oct 31 01:15:19.503022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.503001 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerStarted","Data":"284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17"} Oct 31 01:15:19.504704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.504689 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" Oct 31 01:15:19.504818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.504732 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh" event={"ID":"0cbe5cf5-78ac-47de-b445-42e717b05f73","Type":"ContainerDied","Data":"66826ea3c7b40ed2f6c50dd7e368b104f1ab04312e263fc4a9a54c161f91bd9f"} Oct 31 01:15:19.506524 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.506503 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" event={"ID":"64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4","Type":"ContainerDied","Data":"3903493d46015d7e0e3a8fd932d866d788024fb3eb3a1dc45cab2500e5c7e377"} Oct 31 01:15:19.506706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.506622 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66" Oct 31 01:15:19.513293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.513267 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerStarted","Data":"98342521e1eb75a4c426ea6ea252a156fbd8374f0fe158c64eee6db87c07791b"} Oct 31 01:15:19.523514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.523490 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:19.524904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.524879 5011 scope.go:122] "RemoveContainer" containerID="89a36aa8562866ae3005ca5de21ca1c232b6d27c680c3d849a61b7271779f44a" Oct 31 01:15:19.525013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.524973 5011 generic.go:356] "Generic (PLEG): container finished" podID="0202194e-2baf-429f-a03b-e50d04d3c286" containerID="8200af195526b7cc03e0f06c46ef98cd2db04b31886521bdd38f1c2f4fc66c05" exitCode=60 Oct 31 01:15:19.525096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.525077 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerDied","Data":"8200af195526b7cc03e0f06c46ef98cd2db04b31886521bdd38f1c2f4fc66c05"} Oct 31 01:15:19.525459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.525442 5011 scope.go:122] "RemoveContainer" containerID="8200af195526b7cc03e0f06c46ef98cd2db04b31886521bdd38f1c2f4fc66c05" Oct 31 01:15:19.525720 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:19.525699 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h_odh-ai-gateway-infra(0202194e-2baf-429f-a03b-e50d04d3c286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" Oct 31 01:15:19.527739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.527711 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" Oct 31 01:15:19.527739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.527724 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp" event={"ID":"a9efcb13-2067-4675-8a03-d131feaeefe2","Type":"ContainerDied","Data":"7a2bc4cab93472459e2e677feeb32e25397cc7580b8b90e97cd035313d460872"} Oct 31 01:15:19.536153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.536096 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerStarted","Data":"b9ad6c3063ebc2bdcdf3bf207a6c31c35fdffb55b7983c6104f7b14accbf4243"} Oct 31 01:15:19.537814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.537793 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" Oct 31 01:15:19.538121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.538065 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn" event={"ID":"65df237a-12c1-431a-b296-c6511639ac6a","Type":"ContainerDied","Data":"fbf6d474748b5fa1ed0622e868a17c2d4533aa9f8d9ac59aa331725f44845149"} Oct 31 01:15:19.540650 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.540629 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" event={"ID":"27a08fcb-2bf0-4856-ae94-672abb421881","Type":"ContainerDied","Data":"1f88494742a0494999dbd5995cd3940adcaf514cd3e0cc4316581c5b6f592432"} Oct 31 01:15:19.540733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.540692 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw" Oct 31 01:15:19.560114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.560080 5011 scope.go:122] "RemoveContainer" containerID="ca11d5f0b4c44ba864b5bb99cc02ade8517243a9aaa92a8386cd63aa7c2269cf" Oct 31 01:15:19.583805 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.583787 5011 scope.go:122] "RemoveContainer" containerID="ef9f1c199321aa238cf57fbc62a01caed021c3da66c1431e13b29860a95c18c4" Oct 31 01:15:19.601448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.601430 5011 scope.go:122] "RemoveContainer" containerID="b64b1fb76403887653476de13d0eb6b6e613f149245d1e5e8fcca456cb7e7539" Oct 31 01:15:19.607756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.607729 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:19.622041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.622011 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890155-x6x66"] Oct 31 01:15:19.633017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.632992 5011 scope.go:122] "RemoveContainer" containerID="ba70a63af5fc8f6e06c2ad1e15de983933d6e3ed20c775ce257c1eab20e7311e" Oct 31 01:15:19.645045 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.645018 5011 scope.go:122] "RemoveContainer" containerID="ddc3c4124342bb8fd01801f21a207c7e7711a0e0bf9831a7dfe817c9140fb410" Oct 31 01:15:19.656311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.656284 5011 scope.go:122] "RemoveContainer" containerID="cd0dacd140d8c02f63e05be3928ab8e20f0fd15016adca37f84aa28dc5d56f2b" Oct 31 01:15:19.684345 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.684317 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" path="/var/lib/kubelet/pods/64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4/volumes" Oct 31 01:15:19.821921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.821883 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:19.826395 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.826367 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99a5c2-29890155-nbxpp"] Oct 31 01:15:19.859527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.859499 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:19.866397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:19.866370 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-36af31-29890155-xgkgh"] Oct 31 01:15:20.009468 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.009432 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:20.014202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.014173 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:20.019010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.018984 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-476a67-29890155-5nfsr"] Oct 31 01:15:20.262244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.252572 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:20.263493 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.263464 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b024b5-29890155-hg9kt"] Oct 31 01:15:20.274797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.274715 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:20.297397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.295102 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:20.302395 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.302368 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-4207cd-29890155-tlqgw"] Oct 31 01:15:20.315720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.315692 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:20.323247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.322475 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b8fd5a-29890155-85jwn"] Oct 31 01:15:20.379505 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:20.379472 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26caeb57_fce5_4ac5_adb9_ce621d9f2be4.slice/crio-284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:15:20.407510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.407480 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:20.479830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.479797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\" (UniqueName: \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\") pod \"64aa2b3a-3cf4-4833-9158-ee0669837f86\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " Oct 31 01:15:20.480021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.480004 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\") pod \"64aa2b3a-3cf4-4833-9158-ee0669837f86\" (UID: \"64aa2b3a-3cf4-4833-9158-ee0669837f86\") " Oct 31 01:15:20.480353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.480328 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca" pod "64aa2b3a-3cf4-4833-9158-ee0669837f86" (UID: "64aa2b3a-3cf4-4833-9158-ee0669837f86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.480637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.480623 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa2b3a-3cf4-4833-9158-ee0669837f86-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.481995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.481972 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk" pod "64aa2b3a-3cf4-4833-9158-ee0669837f86" (UID: "64aa2b3a-3cf4-4833-9158-ee0669837f86"). InnerVolumeSpecName "kube-api-access-vhbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.551540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.551510 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" Oct 31 01:15:20.551703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.551548 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp" event={"ID":"64aa2b3a-3cf4-4833-9158-ee0669837f86","Type":"ContainerDied","Data":"17a413f95863636b4eaf66067d4c6c066b770b13da741da9fc83cf8d47a989e4"} Oct 31 01:15:20.551703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.551629 5011 scope.go:122] "RemoveContainer" containerID="21eb1964e7aa86c583b756b9dce101456a8af5cab08837ee4e9c7e9ce12a6942" Oct 31 01:15:20.554057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.554027 5011 generic.go:356] "Generic (PLEG): container finished" podID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerID="284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17" exitCode=60 Oct 31 01:15:20.554182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.554092 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerDied","Data":"284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17"} Oct 31 01:15:20.554608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.554585 5011 scope.go:122] "RemoveContainer" containerID="284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17" Oct 31 01:15:20.554949 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:20.554922 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp_odh-ai-gateway-infra(26caeb57-fce5-4ac5-adb9-ce621d9f2be4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" Oct 31 01:15:20.564107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.564074 5011 generic.go:356] "Generic (PLEG): container finished" podID="149338af-dbda-433a-b141-dd78426fde67" containerID="98342521e1eb75a4c426ea6ea252a156fbd8374f0fe158c64eee6db87c07791b" exitCode=60 Oct 31 01:15:20.564238 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.564147 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerDied","Data":"98342521e1eb75a4c426ea6ea252a156fbd8374f0fe158c64eee6db87c07791b"} Oct 31 01:15:20.564603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.564580 5011 scope.go:122] "RemoveContainer" containerID="98342521e1eb75a4c426ea6ea252a156fbd8374f0fe158c64eee6db87c07791b" Oct 31 01:15:20.564941 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:20.564917 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx_odh-ai-gateway-infra(149338af-dbda-433a-b141-dd78426fde67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" podUID="149338af-dbda-433a-b141-dd78426fde67" Oct 31 01:15:20.565456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.565440 5011 scope.go:122] "RemoveContainer" containerID="1caf6b4e57f1576cfab18c271477dc11440bbb63464d978b8e40d99f6ff762ed" Oct 31 01:15:20.569364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.569344 5011 generic.go:356] "Generic (PLEG): container finished" podID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerID="b9ad6c3063ebc2bdcdf3bf207a6c31c35fdffb55b7983c6104f7b14accbf4243" exitCode=60 Oct 31 01:15:20.569457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.569405 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerDied","Data":"b9ad6c3063ebc2bdcdf3bf207a6c31c35fdffb55b7983c6104f7b14accbf4243"} Oct 31 01:15:20.569795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.569774 5011 scope.go:122] "RemoveContainer" containerID="b9ad6c3063ebc2bdcdf3bf207a6c31c35fdffb55b7983c6104f7b14accbf4243" Oct 31 01:15:20.570056 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:20.570034 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl_odh-ai-gateway-infra(170cd7e9-031e-491e-8b97-c71acf4703e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" Oct 31 01:15:20.581655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.581628 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbhk\" (UniqueName: \"kubernetes.io/projected/64aa2b3a-3cf4-4833-9158-ee0669837f86-kube-api-access-vhbhk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.585898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.582462 5011 scope.go:122] "RemoveContainer" containerID="ebfcd8e2dea5f7e14f71a295401844aa310f60127bf080c54d34fce609e8c846" Oct 31 01:15:20.601671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.601529 5011 scope.go:122] "RemoveContainer" containerID="12a001016c56e167393bbdd6c5073fffad55ad357d084a232f79626fbd5f068d" Oct 31 01:15:20.611284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.611246 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:20.618692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.618650 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-bcdb4e-29890155-lxrwp"] Oct 31 01:15:20.697942 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.697908 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:20.729335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.729304 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:20.802321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.802289 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:20.825652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.825626 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:20.886898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.886866 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\" (UniqueName: \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\") pod \"8356a45b-2097-440f-b6ae-698d125ab90a\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " Oct 31 01:15:20.887103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.887082 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\" (UniqueName: \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\") pod \"0202194e-2baf-429f-a03b-e50d04d3c286\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " Oct 31 01:15:20.887462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.887431 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\") pod \"0202194e-2baf-429f-a03b-e50d04d3c286\" (UID: \"0202194e-2baf-429f-a03b-e50d04d3c286\") " Oct 31 01:15:20.887549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.887491 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\") pod \"8356a45b-2097-440f-b6ae-698d125ab90a\" (UID: \"8356a45b-2097-440f-b6ae-698d125ab90a\") " Oct 31 01:15:20.890784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.887967 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca" pod "8356a45b-2097-440f-b6ae-698d125ab90a" (UID: "8356a45b-2097-440f-b6ae-698d125ab90a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.890905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.889735 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q" pod "8356a45b-2097-440f-b6ae-698d125ab90a" (UID: "8356a45b-2097-440f-b6ae-698d125ab90a"). InnerVolumeSpecName "kube-api-access-52t5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.890905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.889778 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca" pod "0202194e-2baf-429f-a03b-e50d04d3c286" (UID: "0202194e-2baf-429f-a03b-e50d04d3c286"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.892193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.892169 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m" pod "0202194e-2baf-429f-a03b-e50d04d3c286" (UID: "0202194e-2baf-429f-a03b-e50d04d3c286"). InnerVolumeSpecName "kube-api-access-82r4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.976946 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.976859 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:20.992559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.992527 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82r4m\" (UniqueName: \"kubernetes.io/projected/0202194e-2baf-429f-a03b-e50d04d3c286-kube-api-access-82r4m\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.992559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.992557 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0202194e-2baf-429f-a03b-e50d04d3c286-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.992729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.992567 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356a45b-2097-440f-b6ae-698d125ab90a-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.992729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:20.992576 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52t5q\" (UniqueName: \"kubernetes.io/projected/8356a45b-2097-440f-b6ae-698d125ab90a-kube-api-access-52t5q\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:21.585508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.585477 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" Oct 31 01:15:21.586008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.585504 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h" event={"ID":"0202194e-2baf-429f-a03b-e50d04d3c286","Type":"ContainerDied","Data":"d884744a8170281831d987df2702830210ce668ac0ee9e7e52db3c91e8a1b509"} Oct 31 01:15:21.586008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.585585 5011 scope.go:122] "RemoveContainer" containerID="8200af195526b7cc03e0f06c46ef98cd2db04b31886521bdd38f1c2f4fc66c05" Oct 31 01:15:21.589144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.589112 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" Oct 31 01:15:21.589280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.589174 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck" event={"ID":"8356a45b-2097-440f-b6ae-698d125ab90a","Type":"ContainerDied","Data":"ef4f0e4af695cf8e6d805a8e6af074c595bcef446fc3806e81a7da5318f86535"} Oct 31 01:15:21.602608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.601928 5011 scope.go:122] "RemoveContainer" containerID="546ae4be92c032af4049feb44bc1e3f1066678b7c7c046bab030182d9d700ea7" Oct 31 01:15:21.693770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.693463 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" path="/var/lib/kubelet/pods/01eeafe6-ab6d-452d-ab2e-aefce14b5a22/volumes" Oct 31 01:15:21.694401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.694225 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" path="/var/lib/kubelet/pods/0cbe5cf5-78ac-47de-b445-42e717b05f73/volumes" Oct 31 01:15:21.695236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.694907 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" path="/var/lib/kubelet/pods/27a08fcb-2bf0-4856-ae94-672abb421881/volumes" Oct 31 01:15:21.698556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.695951 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" path="/var/lib/kubelet/pods/64aa2b3a-3cf4-4833-9158-ee0669837f86/volumes" Oct 31 01:15:21.698556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.696558 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65df237a-12c1-431a-b296-c6511639ac6a" path="/var/lib/kubelet/pods/65df237a-12c1-431a-b296-c6511639ac6a/volumes" Oct 31 01:15:21.698556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.697110 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" path="/var/lib/kubelet/pods/a9efcb13-2067-4675-8a03-d131feaeefe2/volumes" Oct 31 01:15:21.698556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.698063 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" path="/var/lib/kubelet/pods/ebdf7ea4-71eb-4f66-9311-c4af74dabb3a/volumes" Oct 31 01:15:21.743164 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.743134 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:21.750497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.749633 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ada0c1-29890155-rj54h"] Oct 31 01:15:21.782371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.782338 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:21.788588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.788559 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f2bdf3-29890155-8vbck"] Oct 31 01:15:21.833902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.833756 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:21.864831 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.864798 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:21.889970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.889940 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:21.907805 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.907771 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\" (UniqueName: \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\") pod \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " Oct 31 01:15:21.907970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.907825 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\") pod \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\" (UID: \"26caeb57-fce5-4ac5-adb9-ce621d9f2be4\") " Oct 31 01:15:21.907970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.907875 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\") pod \"149338af-dbda-433a-b141-dd78426fde67\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " Oct 31 01:15:21.908097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.908058 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\") pod \"170cd7e9-031e-491e-8b97-c71acf4703e5\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " Oct 31 01:15:21.908162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.908110 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\" (UniqueName: \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\") pod \"149338af-dbda-433a-b141-dd78426fde67\" (UID: \"149338af-dbda-433a-b141-dd78426fde67\") " Oct 31 01:15:21.908258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.908166 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\" (UniqueName: \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\") pod \"170cd7e9-031e-491e-8b97-c71acf4703e5\" (UID: \"170cd7e9-031e-491e-8b97-c71acf4703e5\") " Oct 31 01:15:21.910099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.910058 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca" pod "149338af-dbda-433a-b141-dd78426fde67" (UID: "149338af-dbda-433a-b141-dd78426fde67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:21.911329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.911132 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca" pod "26caeb57-fce5-4ac5-adb9-ce621d9f2be4" (UID: "26caeb57-fce5-4ac5-adb9-ce621d9f2be4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:21.911329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.911298 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca" pod "170cd7e9-031e-491e-8b97-c71acf4703e5" (UID: "170cd7e9-031e-491e-8b97-c71acf4703e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:21.913502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.913477 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x" pod "170cd7e9-031e-491e-8b97-c71acf4703e5" (UID: "170cd7e9-031e-491e-8b97-c71acf4703e5"). InnerVolumeSpecName "kube-api-access-xfl9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:21.914128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.914106 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d" pod "149338af-dbda-433a-b141-dd78426fde67" (UID: "149338af-dbda-433a-b141-dd78426fde67"). InnerVolumeSpecName "kube-api-access-4rs2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:21.920384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:21.920361 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd" pod "26caeb57-fce5-4ac5-adb9-ce621d9f2be4" (UID: "26caeb57-fce5-4ac5-adb9-ce621d9f2be4"). InnerVolumeSpecName "kube-api-access-4h6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:22.010611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010573 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rs2d\" (UniqueName: \"kubernetes.io/projected/149338af-dbda-433a-b141-dd78426fde67-kube-api-access-4rs2d\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.010611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010614 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfl9x\" (UniqueName: \"kubernetes.io/projected/170cd7e9-031e-491e-8b97-c71acf4703e5-kube-api-access-xfl9x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.010788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010630 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6qd\" (UniqueName: \"kubernetes.io/projected/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-kube-api-access-4h6qd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.010788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010644 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26caeb57-fce5-4ac5-adb9-ce621d9f2be4-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.010788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010659 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149338af-dbda-433a-b141-dd78426fde67-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.010788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.010671 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170cd7e9-031e-491e-8b97-c71acf4703e5-maas-api-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.096290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.090179 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:15:22.697700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.697650 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" event={"ID":"26caeb57-fce5-4ac5-adb9-ce621d9f2be4","Type":"ContainerDied","Data":"afc3d8254cfdaa6d2816ca70e4f2c5ec00678024b396e2ce6d3b259a29c2080d"} Oct 31 01:15:22.698306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.698275 5011 scope.go:122] "RemoveContainer" containerID="284a3505801bcbd0f2cdd6668530262647bb67f1bd2d22291f1e5b6879a60a17" Oct 31 01:15:22.698549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.698535 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp" Oct 31 01:15:22.719876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.719841 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" event={"ID":"149338af-dbda-433a-b141-dd78426fde67","Type":"ContainerDied","Data":"ccebe9b6f401aab45e58912aa09cbf86aa3e94a1c585121be463afe4f043bd5e"} Oct 31 01:15:22.720154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.720135 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx" Oct 31 01:15:22.744460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.743143 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" event={"ID":"170cd7e9-031e-491e-8b97-c71acf4703e5","Type":"ContainerDied","Data":"3b4561d0515f79fffc6f178f833da10c67ee8749c7d30fdbd523441afa8c3789"} Oct 31 01:15:22.744460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.743286 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl" Oct 31 01:15:22.751296 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.750580 5011 scope.go:122] "RemoveContainer" containerID="98342521e1eb75a4c426ea6ea252a156fbd8374f0fe158c64eee6db87c07791b" Oct 31 01:15:22.764837 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.764814 5011 scope.go:122] "RemoveContainer" containerID="b9ad6c3063ebc2bdcdf3bf207a6c31c35fdffb55b7983c6104f7b14accbf4243" Oct 31 01:15:22.818876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.818844 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:22.828320 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.828295 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-daaf26-29890155-gmlzp"] Oct 31 01:15:22.852434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.852402 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:22.860327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.860305 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890155-qz7zx"] Oct 31 01:15:22.897489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.897452 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:22.902310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:22.902285 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d10721-29890155-lpxcl"] Oct 31 01:15:23.696089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:23.696056 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" path="/var/lib/kubelet/pods/0202194e-2baf-429f-a03b-e50d04d3c286/volumes" Oct 31 01:15:23.696552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:23.696535 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149338af-dbda-433a-b141-dd78426fde67" path="/var/lib/kubelet/pods/149338af-dbda-433a-b141-dd78426fde67/volumes" Oct 31 01:15:23.696976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:23.696956 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" path="/var/lib/kubelet/pods/170cd7e9-031e-491e-8b97-c71acf4703e5/volumes" Oct 31 01:15:23.697746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:23.697724 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" path="/var/lib/kubelet/pods/26caeb57-fce5-4ac5-adb9-ce621d9f2be4/volumes" Oct 31 01:15:23.698398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:23.698336 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" path="/var/lib/kubelet/pods/8356a45b-2097-440f-b6ae-698d125ab90a/volumes" Oct 31 01:15:25.583427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.583393 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10896-e736b4"] Oct 31 01:15:25.584053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584037 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584053 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584064 5011 cpu_manager.go:414] "containerMap: removing container" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584069 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584079 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584084 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584097 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584105 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584115 5011 cpu_manager.go:414] "containerMap: removing container" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584119 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584131 5011 cpu_manager.go:414] "containerMap: removing container" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584136 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584142 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584146 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584153 5011 cpu_manager.go:414] "containerMap: removing container" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584158 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584169 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584176 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584187 5011 cpu_manager.go:414] "containerMap: removing container" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584191 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584198 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584202 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584226 5011 cpu_manager.go:414] "containerMap: removing container" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584232 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584240 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584246 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584266 5011 cpu_manager.go:414] "containerMap: removing container" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584271 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584278 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584282 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584291 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584295 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584306 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584310 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584317 5011 cpu_manager.go:414] "containerMap: removing container" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584321 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584329 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584334 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584339 5011 cpu_manager.go:414] "containerMap: removing container" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584344 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584353 5011 cpu_manager.go:414] "containerMap: removing container" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584361 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584376 5011 cpu_manager.go:414] "containerMap: removing container" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584381 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584387 5011 cpu_manager.go:414] "containerMap: removing container" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584392 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584398 5011 cpu_manager.go:414] "containerMap: removing container" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584402 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584413 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584418 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584424 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584429 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584437 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584442 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584448 5011 cpu_manager.go:414] "containerMap: removing container" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584452 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584460 5011 cpu_manager.go:414] "containerMap: removing container" podUID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" containerName="payload-processing" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584465 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" containerName="payload-processing" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584472 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584476 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584485 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584491 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584504 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584509 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584619 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584629 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584635 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584641 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584646 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584651 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584657 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584663 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584669 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584674 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584678 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584685 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584690 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584695 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584701 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584707 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584715 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="020d2d90-4a77-4a24-a4ac-52b4ae6f1439" containerName="payload-processing" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584723 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584730 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584736 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584742 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584749 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:25.584705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584755 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584761 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584766 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584771 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584775 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584780 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe5cf5-78ac-47de-b445-42e717b05f73" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584785 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584793 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.588317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.584799 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9efcb13-2067-4675-8a03-d131feaeefe2" containerName="cleanup" Oct 31 01:15:25.588704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.588359 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:25.601984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.601960 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-e736b4"] Oct 31 01:15:25.608747 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.608727 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kc7d\"" Oct 31 01:15:25.621890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.621863 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htks7\" (UniqueName: \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\") pod \"test-curl-10896-e736b4\" (UID: \"49538232-1b34-4d72-bbeb-b801e26001b8\") " pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:25.723488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.723353 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htks7\" (UniqueName: \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\") pod \"test-curl-10896-e736b4\" (UID: \"49538232-1b34-4d72-bbeb-b801e26001b8\") " pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:25.747270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.747232 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htks7\" (UniqueName: \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\") pod \"test-curl-10896-e736b4\" (UID: \"49538232-1b34-4d72-bbeb-b801e26001b8\") " pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:25.920304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:25.905509 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:26.369778 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:26.369741 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10896-e736b4"] Oct 31 01:15:26.799919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:26.799829 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-e736b4" event={"ID":"49538232-1b34-4d72-bbeb-b801e26001b8","Type":"ContainerStarted","Data":"314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d"} Oct 31 01:15:26.799919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:26.799873 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10896-e736b4" event={"ID":"49538232-1b34-4d72-bbeb-b801e26001b8","Type":"ContainerStarted","Data":"470fcb1713866c9b64a3ccdd9beeb23fefc5d5a15665f545d80986c64f80c10f"} Oct 31 01:15:26.840452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:26.840403 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10896-e736b4" podStartSLOduration=1.840390035 podStartE2EDuration="1.840390035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:26.83754343 +0000 UTC m=+3395.286631069" watchObservedRunningTime="2026-10-31 01:15:26.840390035 +0000 UTC m=+3395.289477674" Oct 31 01:15:27.854144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:27.854110 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10896-e736b4"] Oct 31 01:15:27.859831 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:27.859799 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10896-e736b4"] Oct 31 01:15:28.093332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.093291 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x"] Oct 31 01:15:28.094165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094126 5011 cpu_manager.go:414] "containerMap: removing container" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" containerName="test-curl-10896-e736b4" Oct 31 01:15:28.094165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094155 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" containerName="test-curl-10896-e736b4" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094176 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094184 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094203 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094225 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0202194e-2baf-429f-a03b-e50d04d3c286" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094248 5011 cpu_manager.go:414] "containerMap: removing container" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094255 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094270 5011 cpu_manager.go:414] "containerMap: removing container" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094277 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094286 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094293 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356a45b-2097-440f-b6ae-698d125ab90a" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094325 5011 cpu_manager.go:414] "containerMap: removing container" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094332 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094355 5011 cpu_manager.go:414] "containerMap: removing container" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094362 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094373 5011 cpu_manager.go:414] "containerMap: removing container" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:28.094452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094380 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094552 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="170cd7e9-031e-491e-8b97-c71acf4703e5" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094575 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" containerName="test-curl-10896-e736b4" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094589 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdf7ea4-71eb-4f66-9311-c4af74dabb3a" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094599 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="26caeb57-fce5-4ac5-adb9-ce621d9f2be4" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094611 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eeafe6-ab6d-452d-ab2e-aefce14b5a22" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094622 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cd0ce7-e515-4fd6-bff3-0a975b8d3cd4" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094632 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="149338af-dbda-433a-b141-dd78426fde67" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094641 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="65df237a-12c1-431a-b296-c6511639ac6a" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094651 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a08fcb-2bf0-4856-ae94-672abb421881" containerName="cleanup" Oct 31 01:15:28.095133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.094661 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="64aa2b3a-3cf4-4833-9158-ee0669837f86" containerName="cleanup" Oct 31 01:15:28.098448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.098420 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.116352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.116329 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:28.127298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.127264 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x"] Oct 31 01:15:28.128609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.128592 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:15:28.196136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.196111 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.196312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.196290 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24bk\" (UniqueName: \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.196521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.196504 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.297806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.297777 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24bk\" (UniqueName: \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.297958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.297922 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.297958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.297952 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.298576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.298550 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.300549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.300525 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.321700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.321675 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24bk\" (UniqueName: \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\") pod \"maas-api-revoke-sub-e4162c32-z4s8x\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.418856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.418789 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:28.739424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.739394 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x"] Oct 31 01:15:28.740630 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:28.740605 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc6754a_0245_470d_a3fd_d61d52de6ab4.slice/crio-87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59 WatchSource:0}: Error finding container 87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59: Status 404 returned error can't find the container with id 87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59 Oct 31 01:15:28.816991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.816959 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" event={"ID":"4dc6754a-0245-470d-a3fd-d61d52de6ab4","Type":"ContainerStarted","Data":"87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59"} Oct 31 01:15:28.817258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:28.817187 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10896-e736b4" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" containerName="test-curl-10896-e736b4" containerID="cri-o://314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d" gracePeriod=2 Oct 31 01:15:28.832160 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:28.832118 5011 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10896-e736b4\" is forbidden: User \"system:node:ip-10-0-132-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-66.ec2.internal' and this object" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:29.826636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:29.826604 5011 generic.go:356] "Generic (PLEG): container finished" podID="4dc6754a-0245-470d-a3fd-d61d52de6ab4" containerID="3bb9d755549bc0d922ecedcf604e1882587fb24be7314cfe4549bd24cf7a1506" exitCode=0 Oct 31 01:15:29.826964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:29.826691 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" event={"ID":"4dc6754a-0245-470d-a3fd-d61d52de6ab4","Type":"ContainerDied","Data":"3bb9d755549bc0d922ecedcf604e1882587fb24be7314cfe4549bd24cf7a1506"} Oct 31 01:15:31.016959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.016934 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:31.057825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.057803 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:31.142322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.142249 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\") pod \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " Oct 31 01:15:31.142458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.142376 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\" (UniqueName: \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\") pod \"49538232-1b34-4d72-bbeb-b801e26001b8\" (UID: \"49538232-1b34-4d72-bbeb-b801e26001b8\") " Oct 31 01:15:31.142458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.142407 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\") pod \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " Oct 31 01:15:31.142534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.142494 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\" (UniqueName: \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\") pod \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\" (UID: \"4dc6754a-0245-470d-a3fd-d61d52de6ab4\") " Oct 31 01:15:31.142674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.142636 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca" pod "4dc6754a-0245-470d-a3fd-d61d52de6ab4" (UID: "4dc6754a-0245-470d-a3fd-d61d52de6ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:31.143056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.143026 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:31.144529 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.144506 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7" pod "49538232-1b34-4d72-bbeb-b801e26001b8" (UID: "49538232-1b34-4d72-bbeb-b801e26001b8"). InnerVolumeSpecName "kube-api-access-htks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:31.144620 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.144573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca" pod "4dc6754a-0245-470d-a3fd-d61d52de6ab4" (UID: "4dc6754a-0245-470d-a3fd-d61d52de6ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:31.144668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.144646 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk" pod "4dc6754a-0245-470d-a3fd-d61d52de6ab4" (UID: "4dc6754a-0245-470d-a3fd-d61d52de6ab4"). InnerVolumeSpecName "kube-api-access-h24bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:31.244515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.244481 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htks7\" (UniqueName: \"kubernetes.io/projected/49538232-1b34-4d72-bbeb-b801e26001b8-kube-api-access-htks7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:31.244515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.244513 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc6754a-0245-470d-a3fd-d61d52de6ab4-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:31.244706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.244527 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24bk\" (UniqueName: \"kubernetes.io/projected/4dc6754a-0245-470d-a3fd-d61d52de6ab4-kube-api-access-h24bk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:31.686099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.686065 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49538232-1b34-4d72-bbeb-b801e26001b8" path="/var/lib/kubelet/pods/49538232-1b34-4d72-bbeb-b801e26001b8/volumes" Oct 31 01:15:31.846993 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.846967 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" Oct 31 01:15:31.847160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.846997 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x" event={"ID":"4dc6754a-0245-470d-a3fd-d61d52de6ab4","Type":"ContainerDied","Data":"87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59"} Oct 31 01:15:31.847160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.847052 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87565cd8a4685419b30fc9e4415314798aeb6733ae3d659f52aa524d39989e59" Oct 31 01:15:31.848853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.848825 5011 generic.go:356] "Generic (PLEG): container finished" podID="49538232-1b34-4d72-bbeb-b801e26001b8" containerID="314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d" exitCode=137 Oct 31 01:15:31.848999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.848865 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10896-e736b4" Oct 31 01:15:31.849327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.848879 5011 scope.go:122] "RemoveContainer" containerID="314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d" Oct 31 01:15:31.859735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.859717 5011 scope.go:122] "RemoveContainer" containerID="314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d" Oct 31 01:15:31.860006 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:31.859983 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d\": container with ID starting with 314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d not found: ID does not exist" containerID="314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d" Oct 31 01:15:31.860127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:31.860014 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d"} err="failed to get container status \"314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d\": rpc error: code = NotFound desc = could not find container \"314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d\": container with ID starting with 314fea700df24071b79dada1c0bf8bd26efee856b3c78271b6552100092a951d not found: ID does not exist" Oct 31 01:15:31.946689 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:15:31.946608 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:15:33.679526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.679461 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" containerID="cri-o://f3d222eb8332c1b60cb700c3ed50ca14a62df72330ed52ca10365bda1384e7f2" gracePeriod=45 Oct 31 01:15:33.703577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.703541 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" containerID="cri-o://b329fd9d0c466b79230d3d133524d854d78d974287b0f4f5766b6f97484dcfbe" gracePeriod=45 Oct 31 01:15:33.874491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.874441 5011 generic.go:356] "Generic (PLEG): container finished" podID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerID="b329fd9d0c466b79230d3d133524d854d78d974287b0f4f5766b6f97484dcfbe" exitCode=0 Oct 31 01:15:33.874758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.874511 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" event={"ID":"1d447db4-0d26-47da-b86c-efdaa1bc173c","Type":"ContainerDied","Data":"b329fd9d0c466b79230d3d133524d854d78d974287b0f4f5766b6f97484dcfbe"} Oct 31 01:15:33.876501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.876474 5011 generic.go:356] "Generic (PLEG): container finished" podID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerID="f3d222eb8332c1b60cb700c3ed50ca14a62df72330ed52ca10365bda1384e7f2" exitCode=0 Oct 31 01:15:33.876652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:33.876539 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" event={"ID":"733ad5e7-d7fb-4983-9454-57ac7a885359","Type":"ContainerDied","Data":"f3d222eb8332c1b60cb700c3ed50ca14a62df72330ed52ca10365bda1384e7f2"} Oct 31 01:15:34.012661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.012628 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 01:15:34.048978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.048892 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 01:15:34.080646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080609 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\" (UniqueName: \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.080835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080725 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.080835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080770 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.080835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080792 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.080835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080813 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\" (UniqueName: \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.080835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080831 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080895 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.080974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\") pod \"1d447db4-0d26-47da-b86c-efdaa1bc173c\" (UID: \"1d447db4-0d26-47da-b86c-efdaa1bc173c\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.081001 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\" (UniqueName: \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.081038 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.081130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.081060 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\") pod \"733ad5e7-d7fb-4983-9454-57ac7a885359\" (UID: \"733ad5e7-d7fb-4983-9454-57ac7a885359\") " Oct 31 01:15:34.081913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.081885 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.089125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.082307 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.089125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.082346 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.089125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.082528 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.089125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.082737 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.089676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.089629 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.090623 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.090597 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "kube-api-access-dxj75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:34.091303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.091274 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:34.115493 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.113798 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm" pod "1d447db4-0d26-47da-b86c-efdaa1bc173c" (UID: "1d447db4-0d26-47da-b86c-efdaa1bc173c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.115493 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.113930 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:34.115493 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.114074 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "kube-api-access-45mwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:34.127409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.127376 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm" pod "733ad5e7-d7fb-4983-9454-57ac7a885359" (UID: "733ad5e7-d7fb-4983-9454-57ac7a885359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:34.183460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183423 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183461 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183476 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183492 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxj75\" (UniqueName: \"kubernetes.io/projected/1d447db4-0d26-47da-b86c-efdaa1bc173c-kube-api-access-dxj75\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183506 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183519 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d447db4-0d26-47da-b86c-efdaa1bc173c-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183531 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183543 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d447db4-0d26-47da-b86c-efdaa1bc173c-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183555 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733ad5e7-d7fb-4983-9454-57ac7a885359-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183567 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183581 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733ad5e7-d7fb-4983-9454-57ac7a885359-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.183713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.183594 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mwr\" (UniqueName: \"kubernetes.io/projected/733ad5e7-d7fb-4983-9454-57ac7a885359-kube-api-access-45mwr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:34.886437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.886406 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" Oct 31 01:15:34.886940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.886447 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw" event={"ID":"1d447db4-0d26-47da-b86c-efdaa1bc173c","Type":"ContainerDied","Data":"eebc444b34bddb912a4c727f3952409c9c9986970cbf70e0ab5f52fae6284960"} Oct 31 01:15:34.886940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.886500 5011 scope.go:122] "RemoveContainer" containerID="b329fd9d0c466b79230d3d133524d854d78d974287b0f4f5766b6f97484dcfbe" Oct 31 01:15:34.888503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.888483 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" Oct 31 01:15:34.888701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.888516 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7" event={"ID":"733ad5e7-d7fb-4983-9454-57ac7a885359","Type":"ContainerDied","Data":"63fda5ec0727e50aa03c89158d3bd9f32c5f16e3f1aa85afe104b815ed77f25c"} Oct 31 01:15:34.901338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.901315 5011 scope.go:122] "RemoveContainer" containerID="f3d222eb8332c1b60cb700c3ed50ca14a62df72330ed52ca10365bda1384e7f2" Oct 31 01:15:34.942161 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.941836 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 01:15:34.953537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.953478 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/simulator-w4-0d8bae-kserve-58fd7568c5-7rcgw"] Oct 31 01:15:34.980333 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.980310 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 01:15:34.984744 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:34.984722 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0d8bae/distinct-w4-0d8bae-kserve-59977f8cc7-lrfj7"] Oct 31 01:15:35.684429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:35.684397 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" path="/var/lib/kubelet/pods/1d447db4-0d26-47da-b86c-efdaa1bc173c/volumes" Oct 31 01:15:35.684955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:35.684931 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" path="/var/lib/kubelet/pods/733ad5e7-d7fb-4983-9454-57ac7a885359/volumes" Oct 31 01:15:38.451080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451048 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt"] Oct 31 01:15:38.451701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451680 5011 cpu_manager.go:414] "containerMap: removing container" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" Oct 31 01:15:38.451701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451700 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" Oct 31 01:15:38.451825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451717 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4dc6754a-0245-470d-a3fd-d61d52de6ab4" containerName="revoke-keys" Oct 31 01:15:38.451825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451723 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc6754a-0245-470d-a3fd-d61d52de6ab4" containerName="revoke-keys" Oct 31 01:15:38.451825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451757 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" Oct 31 01:15:38.451825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451765 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" Oct 31 01:15:38.452044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451865 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="733ad5e7-d7fb-4983-9454-57ac7a885359" containerName="main" Oct 31 01:15:38.452044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451877 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc6754a-0245-470d-a3fd-d61d52de6ab4" containerName="revoke-keys" Oct 31 01:15:38.452044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.451890 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d447db4-0d26-47da-b86c-efdaa1bc173c" containerName="main" Oct 31 01:15:38.457417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.457394 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.470533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.470510 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt"] Oct 31 01:15:38.473029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.473005 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:38.473438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.473419 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:15:38.549517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.549480 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.549745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.549656 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q952h\" (UniqueName: \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.549745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.549698 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.651623 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.651587 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.651813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.651720 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q952h\" (UniqueName: \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.651813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.651759 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.652408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.652379 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.654279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.654254 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.675401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.675376 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q952h\" (UniqueName: \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\") pod \"maas-api-revoke-sub-d4d28f0c-tw8rt\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:38.775255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:38.775163 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:39.090810 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:39.090782 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f05b8af_531c_4ee1_94e2_11cba175a463.slice/crio-0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2 WatchSource:0}: Error finding container 0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2: Status 404 returned error can't find the container with id 0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2 Oct 31 01:15:39.091452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:39.091426 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt"] Oct 31 01:15:39.960126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:39.960053 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" event={"ID":"0f05b8af-531c-4ee1-94e2-11cba175a463","Type":"ContainerStarted","Data":"b3e3a3c1a9c3da396961dae043b7bb74ce49f9adf97d6a7f88786f4cadf2a8f8"} Oct 31 01:15:39.960126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:39.960123 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" event={"ID":"0f05b8af-531c-4ee1-94e2-11cba175a463","Type":"ContainerStarted","Data":"0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2"} Oct 31 01:15:40.970413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:40.970379 5011 generic.go:356] "Generic (PLEG): container finished" podID="0f05b8af-531c-4ee1-94e2-11cba175a463" containerID="b3e3a3c1a9c3da396961dae043b7bb74ce49f9adf97d6a7f88786f4cadf2a8f8" exitCode=0 Oct 31 01:15:40.970897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:40.970462 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" event={"ID":"0f05b8af-531c-4ee1-94e2-11cba175a463","Type":"ContainerDied","Data":"b3e3a3c1a9c3da396961dae043b7bb74ce49f9adf97d6a7f88786f4cadf2a8f8"} Oct 31 01:15:41.143010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.142974 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:41.189453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.189420 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\" (UniqueName: \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\") pod \"0f05b8af-531c-4ee1-94e2-11cba175a463\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " Oct 31 01:15:41.189675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.189471 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\") pod \"0f05b8af-531c-4ee1-94e2-11cba175a463\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " Oct 31 01:15:41.189675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.189563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\") pod \"0f05b8af-531c-4ee1-94e2-11cba175a463\" (UID: \"0f05b8af-531c-4ee1-94e2-11cba175a463\") " Oct 31 01:15:41.189897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.189857 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca" pod "0f05b8af-531c-4ee1-94e2-11cba175a463" (UID: "0f05b8af-531c-4ee1-94e2-11cba175a463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:41.190303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.190278 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:41.192275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.192244 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h" pod "0f05b8af-531c-4ee1-94e2-11cba175a463" (UID: "0f05b8af-531c-4ee1-94e2-11cba175a463"). InnerVolumeSpecName "kube-api-access-q952h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:41.193828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.193799 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca" pod "0f05b8af-531c-4ee1-94e2-11cba175a463" (UID: "0f05b8af-531c-4ee1-94e2-11cba175a463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:41.291440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.291368 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q952h\" (UniqueName: \"kubernetes.io/projected/0f05b8af-531c-4ee1-94e2-11cba175a463-kube-api-access-q952h\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:41.291440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.291395 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f05b8af-531c-4ee1-94e2-11cba175a463-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:41.979891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.979807 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" Oct 31 01:15:41.979891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.979843 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt" event={"ID":"0f05b8af-531c-4ee1-94e2-11cba175a463","Type":"ContainerDied","Data":"0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2"} Oct 31 01:15:41.980448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:41.979896 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0905e734d27a93f91480a4c729966567c7d0c2c104e8065c81761ba5806b2ac2" Oct 31 01:15:43.061651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.061612 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:15:43.062674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.062646 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0f05b8af-531c-4ee1-94e2-11cba175a463" containerName="revoke-keys" Oct 31 01:15:43.062674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.062672 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f05b8af-531c-4ee1-94e2-11cba175a463" containerName="revoke-keys" Oct 31 01:15:43.062897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.062861 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f05b8af-531c-4ee1-94e2-11cba175a463" containerName="revoke-keys" Oct 31 01:15:43.108424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.108395 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:15:43.140322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.108622 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.151541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.151509 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d68696-openshift-default-dockercfg-z9khx\"" Oct 31 01:15:43.231673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.231629 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.231967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.231941 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh88x\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232098 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232263 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232445 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232574 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232560 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232717 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.232859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.232844 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.233022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.233004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.347739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.347699 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348032 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348156 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348342 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348326 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348616 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348807 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.348954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.348940 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh88x\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.353192 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.349063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.355511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.349926 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.355511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.350157 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.355511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.353671 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.355511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.354284 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.355511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.354896 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.362557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.362530 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.362967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.362933 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.382703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.382673 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.385660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.385605 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh88x\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\") pod \"e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.566408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.566369 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:43.917368 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:43.917333 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38024f7b_d715_44ce_a5f2_363d5473b559.slice/crio-f0d7eb392b9395668782ce9957cfa3a256f31fc7372408642d0bbc633cdc7dda WatchSource:0}: Error finding container f0d7eb392b9395668782ce9957cfa3a256f31fc7372408642d0bbc633cdc7dda: Status 404 returned error can't find the container with id f0d7eb392b9395668782ce9957cfa3a256f31fc7372408642d0bbc633cdc7dda Oct 31 01:15:43.919948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.919908 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:15:43.920878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.920841 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:15:43.920997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.920923 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:15:43.920997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.920970 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:15:43.998983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:43.998955 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerStarted","Data":"f0d7eb392b9395668782ce9957cfa3a256f31fc7372408642d0bbc633cdc7dda"} Oct 31 01:15:45.008150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:45.008108 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerStarted","Data":"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb"} Oct 31 01:15:45.072157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:45.072092 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podStartSLOduration=2.072071527 podStartE2EDuration="2.072071527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:45.06955926 +0000 UTC m=+3413.518646900" watchObservedRunningTime="2026-10-31 01:15:45.072071527 +0000 UTC m=+3413.521159168" Oct 31 01:15:45.567202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:45.567158 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:45.573595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:45.573564 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:46.016455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:46.016373 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:46.017361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:46.017341 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:15:49.534048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.534014 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:15:49.538634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.538607 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.557130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.557092 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.557314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.557192 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bl7d\" (UniqueName: \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.566498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.566468 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:15:49.578933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.578904 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d68696-dockercfg-cwjvt\"" Oct 31 01:15:49.606839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.606797 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d68696\"" Oct 31 01:15:49.660340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.660301 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.660646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.660622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bl7d\" (UniqueName: \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.668947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.662006 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.755947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.755871 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bl7d\" (UniqueName: \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\") pod \"payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:49.865110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:49.865070 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:50.181244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:50.181197 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:15:50.182512 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:50.182482 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1f8bc1_f389_4ada_9356_b73b126bbaf3.slice/crio-c6ddaf668e1918d8d0a082ff3ef20ed2cb5343eddb9d43957cc5903e68267254 WatchSource:0}: Error finding container c6ddaf668e1918d8d0a082ff3ef20ed2cb5343eddb9d43957cc5903e68267254: Status 404 returned error can't find the container with id c6ddaf668e1918d8d0a082ff3ef20ed2cb5343eddb9d43957cc5903e68267254 Oct 31 01:15:51.068567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:51.068522 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" event={"ID":"dd1f8bc1-f389-4ada-9356-b73b126bbaf3","Type":"ContainerStarted","Data":"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a"} Oct 31 01:15:51.069043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:51.068594 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" event={"ID":"dd1f8bc1-f389-4ada-9356-b73b126bbaf3","Type":"ContainerStarted","Data":"c6ddaf668e1918d8d0a082ff3ef20ed2cb5343eddb9d43957cc5903e68267254"} Oct 31 01:15:51.069221 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:51.069172 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:15:51.113231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:51.111625 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" podStartSLOduration=2.111612993 podStartE2EDuration="2.111612993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:51.110006105 +0000 UTC m=+3419.559093745" watchObservedRunningTime="2026-10-31 01:15:51.111612993 +0000 UTC m=+3419.560700631" Oct 31 01:15:53.902410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.902377 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t"] Oct 31 01:15:53.906772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.906752 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:53.912945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.912921 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8"] Oct 31 01:15:53.923233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.923108 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:15:53.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.925285 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:53.930226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.927323 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:53.952229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.948281 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t"] Oct 31 01:15:53.960929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.960903 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8"] Oct 31 01:15:53.997256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.997187 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs6g\" (UniqueName: \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:53.997547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.997525 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv92n\" (UniqueName: \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:53.997771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.997754 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:53.997984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.997969 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:53.998128 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.998114 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:53.998350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:53.998322 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.099937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.099905 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs6g\" (UniqueName: \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.100139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.099955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv92n\" (UniqueName: \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.100139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.100008 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.100139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.100059 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.100139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.100076 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.100139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.100127 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.101396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.101358 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.101396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.101386 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.105577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.105550 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.109702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.109674 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.136224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.135275 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv92n\" (UniqueName: \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\") pod \"maas-api-revoke-sub-bab1ca71-fsp9t\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.146229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.142737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs6g\" (UniqueName: \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\") pod \"maas-api-revoke-sub-7a6908e8-brjp8\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.228278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.227427 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:54.282843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.282797 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:54.663146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.663112 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t"] Oct 31 01:15:54.665307 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:54.665261 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd568a9ce_45ab_4ab7_b4b1_88543bb929fd.slice/crio-23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b WatchSource:0}: Error finding container 23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b: Status 404 returned error can't find the container with id 23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b Oct 31 01:15:54.759427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:54.759400 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8"] Oct 31 01:15:54.760034 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:15:54.760005 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0d58b0_12c4_4991_89c2_2b867869829f.slice/crio-2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af WatchSource:0}: Error finding container 2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af: Status 404 returned error can't find the container with id 2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af Oct 31 01:15:55.116793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:55.116752 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" event={"ID":"d568a9ce-45ab-4ab7-b4b1-88543bb929fd","Type":"ContainerStarted","Data":"23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b"} Oct 31 01:15:55.123837 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:55.123675 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" event={"ID":"ba0d58b0-12c4-4991-89c2-2b867869829f","Type":"ContainerStarted","Data":"2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af"} Oct 31 01:15:56.205942 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:56.205839 5011 generic.go:356] "Generic (PLEG): container finished" podID="d568a9ce-45ab-4ab7-b4b1-88543bb929fd" containerID="ee1b42c2c3681f264a9463b9f1467ce808804843d1c0757f32d5adae9108a653" exitCode=0 Oct 31 01:15:56.206464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:56.206080 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" event={"ID":"d568a9ce-45ab-4ab7-b4b1-88543bb929fd","Type":"ContainerDied","Data":"ee1b42c2c3681f264a9463b9f1467ce808804843d1c0757f32d5adae9108a653"} Oct 31 01:15:56.210569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:56.210530 5011 generic.go:356] "Generic (PLEG): container finished" podID="ba0d58b0-12c4-4991-89c2-2b867869829f" containerID="15dea1ac40a104254acfa5578eca1a1684895d6d8266d7d5a88adc9b58226d54" exitCode=0 Oct 31 01:15:56.210723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:56.210697 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" event={"ID":"ba0d58b0-12c4-4991-89c2-2b867869829f","Type":"ContainerDied","Data":"15dea1ac40a104254acfa5578eca1a1684895d6d8266d7d5a88adc9b58226d54"} Oct 31 01:15:57.682622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.682593 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:57.790579 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.790538 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:57.814739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.814641 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\") pod \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " Oct 31 01:15:57.814925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.814767 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\" (UniqueName: \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\") pod \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " Oct 31 01:15:57.814994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.814929 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\") pod \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\" (UID: \"d568a9ce-45ab-4ab7-b4b1-88543bb929fd\") " Oct 31 01:15:57.815845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.815808 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca" pod "d568a9ce-45ab-4ab7-b4b1-88543bb929fd" (UID: "d568a9ce-45ab-4ab7-b4b1-88543bb929fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:57.822657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.822619 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca" pod "d568a9ce-45ab-4ab7-b4b1-88543bb929fd" (UID: "d568a9ce-45ab-4ab7-b4b1-88543bb929fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:57.823152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.823124 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n" pod "d568a9ce-45ab-4ab7-b4b1-88543bb929fd" (UID: "d568a9ce-45ab-4ab7-b4b1-88543bb929fd"). InnerVolumeSpecName "kube-api-access-cv92n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:57.919457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.919416 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\") pod \"ba0d58b0-12c4-4991-89c2-2b867869829f\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " Oct 31 01:15:57.919644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.919569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\" (UniqueName: \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\") pod \"ba0d58b0-12c4-4991-89c2-2b867869829f\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " Oct 31 01:15:57.919644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.919627 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\") pod \"ba0d58b0-12c4-4991-89c2-2b867869829f\" (UID: \"ba0d58b0-12c4-4991-89c2-2b867869829f\") " Oct 31 01:15:57.921104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.921070 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca" pod "ba0d58b0-12c4-4991-89c2-2b867869829f" (UID: "ba0d58b0-12c4-4991-89c2-2b867869829f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:57.924028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.923995 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g" pod "ba0d58b0-12c4-4991-89c2-2b867869829f" (UID: "ba0d58b0-12c4-4991-89c2-2b867869829f"). InnerVolumeSpecName "kube-api-access-lrs6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:57.926135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926106 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca" pod "ba0d58b0-12c4-4991-89c2-2b867869829f" (UID: "ba0d58b0-12c4-4991-89c2-2b867869829f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:57.926464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926439 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:57.926543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926472 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:57.926543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926489 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:57.926543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926503 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrs6g\" (UniqueName: \"kubernetes.io/projected/ba0d58b0-12c4-4991-89c2-2b867869829f-kube-api-access-lrs6g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:57.926543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926516 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0d58b0-12c4-4991-89c2-2b867869829f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:57.926543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:57.926530 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv92n\" (UniqueName: \"kubernetes.io/projected/d568a9ce-45ab-4ab7-b4b1-88543bb929fd-kube-api-access-cv92n\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:15:58.241481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.241438 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:15:58.249835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.249804 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" Oct 31 01:15:58.249997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.249836 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t" event={"ID":"d568a9ce-45ab-4ab7-b4b1-88543bb929fd","Type":"ContainerDied","Data":"23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b"} Oct 31 01:15:58.249997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.249896 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23039e7c5070bcc311914b0fbeb9e65604b2d046784d6aabe51654a01f239d5b" Oct 31 01:15:58.251458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.251435 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" Oct 31 01:15:58.251592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.251460 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8" event={"ID":"ba0d58b0-12c4-4991-89c2-2b867869829f","Type":"ContainerDied","Data":"2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af"} Oct 31 01:15:58.251592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:15:58.251493 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2351dad1bf5c4999f72e2bfe7073edc01b1aa8961c5610a73df6b3657e5f25af" Oct 31 01:16:01.611635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:01.609531 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": read tcp 10.132.0.2:56434->10.132.0.250:15021: read: connection reset by peer" start-of-body= Oct 31 01:16:01.611635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:01.609592 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": read tcp 10.132.0.2:56434->10.132.0.250:15021: read: connection reset by peer" Oct 31 01:16:01.950333 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:16:01.950260 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:16:02.289004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.288901 5011 generic.go:356] "Generic (PLEG): container finished" podID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerID="5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1" exitCode=0 Oct 31 01:16:02.289004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.288976 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerDied","Data":"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1"} Oct 31 01:16:02.289534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.289516 5011 scope.go:122] "RemoveContainer" containerID="5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1" Oct 31 01:16:02.291494 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.291456 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:16:02.291624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.291552 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:16:02.291624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:02.291587 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:16:03.089487 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.089455 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:16:03.246228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.242264 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" podUID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" containerName="payload-pre-processing" containerID="cri-o://0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed" gracePeriod=25 Oct 31 01:16:03.465299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.464092 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerStarted","Data":"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa"} Oct 31 01:16:03.767073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.767045 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:16:03.892649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.892607 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\") pod \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " Oct 31 01:16:03.892854 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.892674 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\" (UniqueName: \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\") pod \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\" (UID: \"7b15e37d-d3af-4e86-978b-a2f1fbc87a61\") " Oct 31 01:16:03.893031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.893003 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume" pod "7b15e37d-d3af-4e86-978b-a2f1fbc87a61" (UID: "7b15e37d-d3af-4e86-978b-a2f1fbc87a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:03.893243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.893220 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:16:03.895040 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.895011 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd" pod "7b15e37d-d3af-4e86-978b-a2f1fbc87a61" (UID: "7b15e37d-d3af-4e86-978b-a2f1fbc87a61"). InnerVolumeSpecName "kube-api-access-d9fgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:03.895141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.895018 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:16:03.895185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.895170 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:16:03.895420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.895391 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 01:16:03.994817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:03.994740 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fgd\" (UniqueName: \"kubernetes.io/projected/7b15e37d-d3af-4e86-978b-a2f1fbc87a61-kube-api-access-d9fgd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:16:04.473469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.473437 5011 generic.go:356] "Generic (PLEG): container finished" podID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" containerID="0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed" exitCode=0 Oct 31 01:16:04.473994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.473476 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" Oct 31 01:16:04.473994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.473521 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" event={"ID":"7b15e37d-d3af-4e86-978b-a2f1fbc87a61","Type":"ContainerDied","Data":"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed"} Oct 31 01:16:04.473994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.473591 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4" event={"ID":"7b15e37d-d3af-4e86-978b-a2f1fbc87a61","Type":"ContainerDied","Data":"2956ac822f4a6727a37a299118457fcfbba24a43965883eead1af49d0ea532f7"} Oct 31 01:16:04.473994 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.473625 5011 scope.go:122] "RemoveContainer" containerID="0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed" Oct 31 01:16:04.485820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.485648 5011 scope.go:122] "RemoveContainer" containerID="0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed" Oct 31 01:16:04.485935 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:16:04.485917 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed\": container with ID starting with 0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed not found: ID does not exist" containerID="0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed" Oct 31 01:16:04.485974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.485943 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed"} err="failed to get container status \"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed\": rpc error: code = NotFound desc = could not find container \"0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed\": container with ID starting with 0bfeb7d9c78d6c6c976b214d40d312ccf0c8996f382ebf27fd7f52e062d056ed not found: ID does not exist" Oct 31 01:16:04.555888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.555854 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:16:04.563558 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.563529 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa97ea-58c4c47655-qlmj4"] Oct 31 01:16:04.895129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.895093 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:16:04.895344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:04.895145 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 01:16:05.684432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:05.684399 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" path="/var/lib/kubelet/pods/7b15e37d-d3af-4e86-978b-a2f1fbc87a61/volumes" Oct 31 01:16:05.895053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:05.895016 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:16:05.895247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:05.895088 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 01:16:06.894823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:06.894783 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:16:06.895286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:06.894851 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 01:16:07.895029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:07.894994 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:16:07.895450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:07.895069 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 01:16:08.899489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:08.899456 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:16:08.899968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:08.899939 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:16:08.900615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:08.900595 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:16:16.723539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:16.723498 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:16:21.034080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.034046 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt"] Oct 31 01:16:21.047322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.047276 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87105eff-lbrxt"] Oct 31 01:16:21.052983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.052952 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6"] Oct 31 01:16:21.058137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.058111 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5530d91c-95bx6"] Oct 31 01:16:21.685340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.685303 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3" path="/var/lib/kubelet/pods/119d4c99-a0dc-4cc6-88a4-3c5ef8adcbd3/volumes" Oct 31 01:16:21.685859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:21.685838 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8aaadbb-9253-4008-bb3f-3f70345f1eef" path="/var/lib/kubelet/pods/b8aaadbb-9253-4008-bb3f-3f70345f1eef/volumes" Oct 31 01:16:23.032974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.032935 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5"] Oct 31 01:16:23.045229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.045190 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64ca7cbf-hh7z5"] Oct 31 01:16:23.685881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.685846 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a" path="/var/lib/kubelet/pods/a83d7b41-f7ac-43ab-a86c-ec3c060e1f2a/volumes" Oct 31 01:16:23.797611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.797580 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks"] Oct 31 01:16:23.798197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798180 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" containerName="payload-pre-processing" Oct 31 01:16:23.798279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798198 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" containerName="payload-pre-processing" Oct 31 01:16:23.798279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798231 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d568a9ce-45ab-4ab7-b4b1-88543bb929fd" containerName="revoke-keys" Oct 31 01:16:23.798279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798237 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d568a9ce-45ab-4ab7-b4b1-88543bb929fd" containerName="revoke-keys" Oct 31 01:16:23.798279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798246 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ba0d58b0-12c4-4991-89c2-2b867869829f" containerName="revoke-keys" Oct 31 01:16:23.798279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798251 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0d58b0-12c4-4991-89c2-2b867869829f" containerName="revoke-keys" Oct 31 01:16:23.798449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798376 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b15e37d-d3af-4e86-978b-a2f1fbc87a61" containerName="payload-pre-processing" Oct 31 01:16:23.798449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798389 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d568a9ce-45ab-4ab7-b4b1-88543bb929fd" containerName="revoke-keys" Oct 31 01:16:23.798449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.798395 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0d58b0-12c4-4991-89c2-2b867869829f" containerName="revoke-keys" Oct 31 01:16:23.803404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.803383 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:23.821604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.821575 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks"] Oct 31 01:16:23.865366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.865339 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:16:23.865742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.865708 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:16:23.941011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.940939 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97fj\" (UniqueName: \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:23.941154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.941131 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:23.941260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:23.941174 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.042843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.042798 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.043336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.042961 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.043336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.043094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97fj\" (UniqueName: \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.043491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.043465 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.045457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.045431 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.067981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.067956 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97fj\" (UniqueName: \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\") pod \"maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.118264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.118185 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:24.435282 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:16:24.435246 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bf32c9_c0dd_4633_abb0_1e78260e9d3f.slice/crio-3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13 WatchSource:0}: Error finding container 3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13: Status 404 returned error can't find the container with id 3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13 Oct 31 01:16:24.437511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.437477 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks"] Oct 31 01:16:24.721809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:24.721599 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" event={"ID":"05bf32c9-c0dd-4633-abb0-1e78260e9d3f","Type":"ContainerStarted","Data":"3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13"} Oct 31 01:16:25.732346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:25.732199 5011 generic.go:356] "Generic (PLEG): container finished" podID="05bf32c9-c0dd-4633-abb0-1e78260e9d3f" containerID="bf5ae48a62e89c0840b288e0cd4a3cc4e9e4b4d6add77f648b892be0dfd30170" exitCode=0 Oct 31 01:16:25.732346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:25.732306 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" event={"ID":"05bf32c9-c0dd-4633-abb0-1e78260e9d3f","Type":"ContainerDied","Data":"bf5ae48a62e89c0840b288e0cd4a3cc4e9e4b4d6add77f648b892be0dfd30170"} Oct 31 01:16:26.918982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:26.918956 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:27.089134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.089101 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\" (UniqueName: \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\") pod \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " Oct 31 01:16:27.089319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.089265 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\") pod \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " Oct 31 01:16:27.089319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.089293 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\") pod \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\" (UID: \"05bf32c9-c0dd-4633-abb0-1e78260e9d3f\") " Oct 31 01:16:27.089692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.089648 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca" pod "05bf32c9-c0dd-4633-abb0-1e78260e9d3f" (UID: "05bf32c9-c0dd-4633-abb0-1e78260e9d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:27.089974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.089954 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:16:27.091432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.091404 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj" pod "05bf32c9-c0dd-4633-abb0-1e78260e9d3f" (UID: "05bf32c9-c0dd-4633-abb0-1e78260e9d3f"). InnerVolumeSpecName "kube-api-access-h97fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:27.091599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.091578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca" pod "05bf32c9-c0dd-4633-abb0-1e78260e9d3f" (UID: "05bf32c9-c0dd-4633-abb0-1e78260e9d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:16:27.191673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.191642 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h97fj\" (UniqueName: \"kubernetes.io/projected/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-kube-api-access-h97fj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:16:27.191673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.191672 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05bf32c9-c0dd-4633-abb0-1e78260e9d3f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:16:27.752088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.752056 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" event={"ID":"05bf32c9-c0dd-4633-abb0-1e78260e9d3f","Type":"ContainerDied","Data":"3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13"} Oct 31 01:16:27.752303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.752096 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks" Oct 31 01:16:27.752303 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:27.752103 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b27ae26756f2390b8b3f6b2fa5a282e6b24887992b471f491187fd8c7025d13" Oct 31 01:16:28.952384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.952350 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:16:28.953224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.953184 5011 cpu_manager.go:414] "containerMap: removing container" podUID="05bf32c9-c0dd-4633-abb0-1e78260e9d3f" containerName="revoke-keys" Oct 31 01:16:28.953331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.953226 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05bf32c9-c0dd-4633-abb0-1e78260e9d3f" containerName="revoke-keys" Oct 31 01:16:28.953416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.953400 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="05bf32c9-c0dd-4633-abb0-1e78260e9d3f" containerName="revoke-keys" Oct 31 01:16:28.957137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.957115 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:28.980911 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.980878 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:16:28.984887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:28.984860 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d68696-serving-cert\"" Oct 31 01:16:29.120627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.120589 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.120876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.120746 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzx6z\" (UniqueName: \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.120876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.120799 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.120876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.120850 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.222605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.222510 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzx6z\" (UniqueName: \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.222781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.222604 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.222781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.222676 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.222925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.222781 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.223415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.223380 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.225420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.225394 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.225530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.225457 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.262116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.259783 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzx6z\" (UniqueName: \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\") pod \"maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.279172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.278382 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:29.603167 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:16:29.603132 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab71b03_c225_41a7_9fa0_291123c90b86.slice/crio-4e63921435dcdbf5c1b639b43e199344cae88b4111d7f4f479f49260fbc1704a WatchSource:0}: Error finding container 4e63921435dcdbf5c1b639b43e199344cae88b4111d7f4f479f49260fbc1704a: Status 404 returned error can't find the container with id 4e63921435dcdbf5c1b639b43e199344cae88b4111d7f4f479f49260fbc1704a Oct 31 01:16:29.603680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.603624 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:16:29.604660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.604631 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:16:29.770748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:29.770710 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" event={"ID":"0ab71b03-c225-41a7-9fa0-291123c90b86","Type":"ContainerStarted","Data":"4e63921435dcdbf5c1b639b43e199344cae88b4111d7f4f479f49260fbc1704a"} Oct 31 01:16:30.780054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:30.780019 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" event={"ID":"0ab71b03-c225-41a7-9fa0-291123c90b86","Type":"ContainerStarted","Data":"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4"} Oct 31 01:16:30.780474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:30.780241 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:30.857151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:30.857087 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" podStartSLOduration=2.44757692 podStartE2EDuration="2.857065887s" totalImagesPullingTime="409.488967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:30.826227396 +0000 UTC m=+3459.275315032" watchObservedRunningTime="2026-10-31 01:16:30.857065887 +0000 UTC m=+3459.306153549" Oct 31 01:16:31.031489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:31.031404 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx"] Oct 31 01:16:31.035423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:31.035394 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc727f3-fdfrx"] Oct 31 01:16:31.685403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:31.685361 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb693b4-564e-4f0e-99a6-816398c3f05a" path="/var/lib/kubelet/pods/dbb693b4-564e-4f0e-99a6-816398c3f05a/volumes" Oct 31 01:16:31.950635 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:16:31.950546 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:16:32.033698 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:32.033665 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft"] Oct 31 01:16:32.044330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:32.044300 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8731ca16-m9dft"] Oct 31 01:16:33.042380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:33.042281 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk"] Oct 31 01:16:33.054282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:33.054248 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac45c7b0-4qpfk"] Oct 31 01:16:33.684274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:33.684238 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c48266-ba3f-4121-af44-cee5dcba2d9c" path="/var/lib/kubelet/pods/35c48266-ba3f-4121-af44-cee5dcba2d9c/volumes" Oct 31 01:16:33.684758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:33.684742 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5417427-0f66-4752-9ccb-fdde69426373" path="/var/lib/kubelet/pods/e5417427-0f66-4752-9ccb-fdde69426373/volumes" Oct 31 01:16:36.794235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:36.794179 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:16:48.663790 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.663750 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:16:48.670616 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.670590 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.697583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.697548 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"rate-test-model-d10721-kserve-self-signed-certs\"" Oct 31 01:16:48.697760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.697663 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"default-dockercfg-fzf6b\"" Oct 31 01:16:48.697760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.697753 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"kube-root-ca.crt\"" Oct 31 01:16:48.697928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.697909 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d10721\"/\"openshift-service-ca.crt\"" Oct 31 01:16:48.714368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.714336 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749512 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749567 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749740 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xfg\" (UniqueName: \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749829 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749906 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.750258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.749938 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xfg\" (UniqueName: \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851582 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851671 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851703 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851856 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.851937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.851890 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.852156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.852125 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.852484 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.852451 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.852590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.852500 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.854382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.854359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.855064 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.855031 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:48.874849 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:48.874826 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xfg\" (UniqueName: \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\") pod \"rate-test-model-d10721-kserve-65bc769fb5-gkkbn\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:49.010330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:49.007462 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:49.393273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:49.393238 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:16:49.393606 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:16:49.393584 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3859949_cb4a_4c68_acba_659c0eea5f16.slice/crio-bf49e93079387076444bbf074fffc603d735a130548a3e9de632f4b747ae89c3 WatchSource:0}: Error finding container bf49e93079387076444bbf074fffc603d735a130548a3e9de632f4b747ae89c3: Status 404 returned error can't find the container with id bf49e93079387076444bbf074fffc603d735a130548a3e9de632f4b747ae89c3 Oct 31 01:16:50.019440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:50.019401 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" event={"ID":"f3859949-cb4a-4c68-acba-659c0eea5f16","Type":"ContainerStarted","Data":"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4"} Oct 31 01:16:50.019440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:50.019448 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" event={"ID":"f3859949-cb4a-4c68-acba-659c0eea5f16","Type":"ContainerStarted","Data":"bf49e93079387076444bbf074fffc603d735a130548a3e9de632f4b747ae89c3"} Oct 31 01:16:50.068415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:50.068351 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" podStartSLOduration=2.068331937 podStartE2EDuration="2.068331937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:50.066751524 +0000 UTC m=+3478.515839166" watchObservedRunningTime="2026-10-31 01:16:50.068331937 +0000 UTC m=+3478.517419582" Oct 31 01:16:51.008459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:51.008418 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:54.722936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:54.722900 5011 scope.go:122] "RemoveContainer" containerID="063d83286ea6ce662e13a14cc5a4108ec399ac7382a570cf434c544f4ee10d8a" Oct 31 01:16:59.008070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:59.008035 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:59.023113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:59.023082 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:16:59.165355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:16:59.165320 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:17:01.948913 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:17:01.948868 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:17:14.032567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:14.032524 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj"] Oct 31 01:17:14.044504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:14.044466 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8f34bb-6pvzj"] Oct 31 01:17:15.335010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.334973 5011 generic.go:356] "Generic (PLEG): container finished" podID="38024f7b-d715-44ce-a5f2-363d5473b559" containerID="d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb" exitCode=0 Oct 31 01:17:15.335590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.335050 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerDied","Data":"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb"} Oct 31 01:17:15.335590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.335509 5011 scope.go:122] "RemoveContainer" containerID="d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb" Oct 31 01:17:15.337320 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.337287 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:17:15.337444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.337360 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:17:15.337444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.337390 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:17:15.684255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:15.684147 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a5fa07-5fbc-416e-b948-2aeea815b482" path="/var/lib/kubelet/pods/25a5fa07-5fbc-416e-b948-2aeea815b482/volumes" Oct 31 01:17:16.017341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:16.017263 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:17:16.385588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:16.385536 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerStarted","Data":"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987"} Oct 31 01:17:16.567173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:16.567134 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:17:16.568429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:16.568397 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:16.568552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:16.568468 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:17.567596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:17.567546 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:17.568107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:17.567626 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:18.557536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:18.557463 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks"] Oct 31 01:17:18.567689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:18.567656 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:18.568070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:18.567709 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:18.569248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:18.569221 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e41918-j42ks"] Oct 31 01:17:19.567051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:19.567009 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:19.567317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:19.567074 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:19.693492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:19.693447 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05bf32c9-c0dd-4633-abb0-1e78260e9d3f" path="/var/lib/kubelet/pods/05bf32c9-c0dd-4633-abb0-1e78260e9d3f/volumes" Oct 31 01:17:19.937817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:19.937720 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:17:20.567863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:20.567756 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:20.567863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:20.567818 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:21.567452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:21.567411 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 31 01:17:21.567917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:21.567494 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 31 01:17:22.572368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:22.572341 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:17:22.572859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:22.572839 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:17:22.573647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:22.573626 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:17:24.032072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:24.032032 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz"] Oct 31 01:17:24.046099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:24.046066 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36859b20-p6fsz"] Oct 31 01:17:24.939021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:24.938976 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" podUID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" containerName="payload-pre-processing" containerID="cri-o://65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a" gracePeriod=25 Oct 31 01:17:25.227716 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.227691 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:17:25.373767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.373726 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\") pod \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " Oct 31 01:17:25.373971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.373822 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\" (UniqueName: \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\") pod \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\" (UID: \"dd1f8bc1-f389-4ada-9356-b73b126bbaf3\") " Oct 31 01:17:25.374173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.374146 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume" pod "dd1f8bc1-f389-4ada-9356-b73b126bbaf3" (UID: "dd1f8bc1-f389-4ada-9356-b73b126bbaf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:25.374416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.374400 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:25.375978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.375954 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d" pod "dd1f8bc1-f389-4ada-9356-b73b126bbaf3" (UID: "dd1f8bc1-f389-4ada-9356-b73b126bbaf3"). InnerVolumeSpecName "kube-api-access-8bl7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:25.476073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.476005 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bl7d\" (UniqueName: \"kubernetes.io/projected/dd1f8bc1-f389-4ada-9356-b73b126bbaf3-kube-api-access-8bl7d\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:25.524264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.524235 5011 generic.go:356] "Generic (PLEG): container finished" podID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" containerID="65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a" exitCode=0 Oct 31 01:17:25.524522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.524330 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" event={"ID":"dd1f8bc1-f389-4ada-9356-b73b126bbaf3","Type":"ContainerDied","Data":"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a"} Oct 31 01:17:25.524522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.524343 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" Oct 31 01:17:25.524522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.524387 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2" event={"ID":"dd1f8bc1-f389-4ada-9356-b73b126bbaf3","Type":"ContainerDied","Data":"c6ddaf668e1918d8d0a082ff3ef20ed2cb5343eddb9d43957cc5903e68267254"} Oct 31 01:17:25.524522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.524420 5011 scope.go:122] "RemoveContainer" containerID="65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a" Oct 31 01:17:25.537173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.537082 5011 scope.go:122] "RemoveContainer" containerID="65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a" Oct 31 01:17:25.537458 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:17:25.537438 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a\": container with ID starting with 65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a not found: ID does not exist" containerID="65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a" Oct 31 01:17:25.537531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.537468 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a"} err="failed to get container status \"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a\": rpc error: code = NotFound desc = could not find container \"65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a\": container with ID starting with 65a3dfc1564ee3268f86a336e73824259f0a06e7b9a127aa90587f1b6726fc3a not found: ID does not exist" Oct 31 01:17:25.578332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.578303 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:17:25.589850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.589825 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68696-664fd77cb7-bwqn2"] Oct 31 01:17:25.685377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.685341 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569cd7b8-fe30-47bc-b063-42a970791cb1" path="/var/lib/kubelet/pods/569cd7b8-fe30-47bc-b063-42a970791cb1/volumes" Oct 31 01:17:25.685895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.685876 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" path="/var/lib/kubelet/pods/dd1f8bc1-f389-4ada-9356-b73b126bbaf3/volumes" Oct 31 01:17:25.897522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.897485 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:17:25.897761 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:25.897729 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" containerID="cri-o://dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73" gracePeriod=30 Oct 31 01:17:26.492607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.492572 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f"] Oct 31 01:17:26.493235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.493219 5011 cpu_manager.go:414] "containerMap: removing container" podUID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" containerName="payload-pre-processing" Oct 31 01:17:26.493288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.493237 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" containerName="payload-pre-processing" Oct 31 01:17:26.493398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.493388 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1f8bc1-f389-4ada-9356-b73b126bbaf3" containerName="payload-pre-processing" Oct 31 01:17:26.498032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.498007 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.515018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.514990 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f"] Oct 31 01:17:26.596152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.596119 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:26.596331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.596261 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:17:26.607949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.607919 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjdd\" (UniqueName: \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.608092 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.607961 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.608254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.608232 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.710151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.710121 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjdd\" (UniqueName: \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.710357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.710170 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.710454 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.710432 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.710892 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.710871 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.713084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.713058 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.733842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.733820 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjdd\" (UniqueName: \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\") pod \"maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:26.813386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:26.813306 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:27.124584 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:17:27.124548 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e580a1_6734_4810_aa37_d7db2739748a.slice/crio-689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8 WatchSource:0}: Error finding container 689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8: Status 404 returned error can't find the container with id 689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8 Oct 31 01:17:27.128271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:27.128242 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f"] Oct 31 01:17:27.547101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:27.547069 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" event={"ID":"72e580a1-6734-4810-aa37-d7db2739748a","Type":"ContainerStarted","Data":"041b4d3eed368e92f9ed318870d113f3e0f99299c91cb62c541db0988286a911"} Oct 31 01:17:27.547541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:27.547129 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" event={"ID":"72e580a1-6734-4810-aa37-d7db2739748a","Type":"ContainerStarted","Data":"689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8"} Oct 31 01:17:27.589222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:27.589161 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" podStartSLOduration=1.5891484340000002 podStartE2EDuration="1.589148434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:27.588171331 +0000 UTC m=+3516.037258970" watchObservedRunningTime="2026-10-31 01:17:27.589148434 +0000 UTC m=+3516.038236102" Oct 31 01:17:28.556812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:28.556778 5011 generic.go:356] "Generic (PLEG): container finished" podID="72e580a1-6734-4810-aa37-d7db2739748a" containerID="041b4d3eed368e92f9ed318870d113f3e0f99299c91cb62c541db0988286a911" exitCode=0 Oct 31 01:17:28.557311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:28.556869 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" event={"ID":"72e580a1-6734-4810-aa37-d7db2739748a","Type":"ContainerDied","Data":"041b4d3eed368e92f9ed318870d113f3e0f99299c91cb62c541db0988286a911"} Oct 31 01:17:29.736612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.736585 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:29.856483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.856449 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\" (UniqueName: \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\") pod \"72e580a1-6734-4810-aa37-d7db2739748a\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " Oct 31 01:17:29.856677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.856553 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\") pod \"72e580a1-6734-4810-aa37-d7db2739748a\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " Oct 31 01:17:29.856925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.856903 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\") pod \"72e580a1-6734-4810-aa37-d7db2739748a\" (UID: \"72e580a1-6734-4810-aa37-d7db2739748a\") " Oct 31 01:17:29.857445 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.857422 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca" pod "72e580a1-6734-4810-aa37-d7db2739748a" (UID: "72e580a1-6734-4810-aa37-d7db2739748a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:29.858675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.858650 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd" pod "72e580a1-6734-4810-aa37-d7db2739748a" (UID: "72e580a1-6734-4810-aa37-d7db2739748a"). InnerVolumeSpecName "kube-api-access-6gjdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:29.859181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.859158 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca" pod "72e580a1-6734-4810-aa37-d7db2739748a" (UID: "72e580a1-6734-4810-aa37-d7db2739748a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:29.965229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.962609 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:29.965229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.962668 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gjdd\" (UniqueName: \"kubernetes.io/projected/72e580a1-6734-4810-aa37-d7db2739748a-kube-api-access-6gjdd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:29.965229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:29.962684 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e580a1-6734-4810-aa37-d7db2739748a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:30.576512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:30.576479 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" Oct 31 01:17:30.576512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:30.576491 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f" event={"ID":"72e580a1-6734-4810-aa37-d7db2739748a","Type":"ContainerDied","Data":"689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8"} Oct 31 01:17:30.576737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:30.576557 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="689c6fb059321afc1d0f2c57971f9462236c3abcbeb5a91887a4120f0c03dee8" Oct 31 01:17:31.244033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.244006 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:17:31.289413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289381 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289484 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289579 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289628 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289634 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:31.289755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289680 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289716 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289771 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:31.289869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289829 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289896 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.289990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\") pod \"23dba605-8432-495d-a997-50c8b6338fa1\" (UID: \"23dba605-8432-495d-a997-50c8b6338fa1\") " Oct 31 01:17:31.290109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.289985 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:31.290109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290077 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:31.290335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290315 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:31.290735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290711 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23dba605-8432-495d-a997-50c8b6338fa1-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.290898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290862 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.290898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290890 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.291083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290907 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.291083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.290922 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.292065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.292036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "kube-api-access-qs4tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:31.292186 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.292153 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:31.292374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.292354 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:17:31.292448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.292371 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy" pod "23dba605-8432-495d-a997-50c8b6338fa1" (UID: "23dba605-8432-495d-a997-50c8b6338fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:31.392654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.392622 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23dba605-8432-495d-a997-50c8b6338fa1-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.392654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.392652 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs4tv\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-kube-api-access-qs4tv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.392654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.392662 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23dba605-8432-495d-a997-50c8b6338fa1-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.392875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.392671 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23dba605-8432-495d-a997-50c8b6338fa1-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:31.588763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.588729 5011 generic.go:356] "Generic (PLEG): container finished" podID="23dba605-8432-495d-a997-50c8b6338fa1" containerID="dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73" exitCode=0 Oct 31 01:17:31.588959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.588779 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" Oct 31 01:17:31.588959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.588815 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerDied","Data":"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73"} Oct 31 01:17:31.588959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.588913 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq" event={"ID":"23dba605-8432-495d-a997-50c8b6338fa1","Type":"ContainerDied","Data":"3f96c6692a3d2915b72aee8c9e8110346eb6ab0ce1e73d5085e511e6e5571b08"} Oct 31 01:17:31.588959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.588942 5011 scope.go:122] "RemoveContainer" containerID="dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73" Oct 31 01:17:31.601399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.601374 5011 scope.go:122] "RemoveContainer" containerID="1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b" Oct 31 01:17:31.611254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.611230 5011 scope.go:122] "RemoveContainer" containerID="dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73" Oct 31 01:17:31.611534 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:17:31.611508 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73\": container with ID starting with dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73 not found: ID does not exist" containerID="dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73" Oct 31 01:17:31.611608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.611535 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73"} err="failed to get container status \"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73\": rpc error: code = NotFound desc = could not find container \"dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73\": container with ID starting with dd0514938198fd2f41b4ae623371bc6b04090ce878c297506af042bb57758e73 not found: ID does not exist" Oct 31 01:17:31.611608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.611551 5011 scope.go:122] "RemoveContainer" containerID="1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b" Oct 31 01:17:31.611773 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:17:31.611758 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b\": container with ID starting with 1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b not found: ID does not exist" containerID="1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b" Oct 31 01:17:31.611814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.611780 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b"} err="failed to get container status \"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b\": rpc error: code = NotFound desc = could not find container \"1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b\": container with ID starting with 1bec435b1e1c38423d6b981134aedb53b59f1c58cb400485ed1b2b4f5d6b8a8b not found: ID does not exist" Oct 31 01:17:31.646484 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.646454 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:17:31.652035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.652016 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e41918-openshift-default-5ccdddb467-7z8gq"] Oct 31 01:17:31.685714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:31.685684 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23dba605-8432-495d-a997-50c8b6338fa1" path="/var/lib/kubelet/pods/23dba605-8432-495d-a997-50c8b6338fa1/volumes" Oct 31 01:17:31.947893 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:17:31.947806 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:17:44.492327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:44.492291 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:17:49.691106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.691069 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94"] Oct 31 01:17:49.692021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.691991 5011 cpu_manager.go:414] "containerMap: removing container" podUID="72e580a1-6734-4810-aa37-d7db2739748a" containerName="revoke-keys" Oct 31 01:17:49.692021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692021 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e580a1-6734-4810-aa37-d7db2739748a" containerName="revoke-keys" Oct 31 01:17:49.692252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692040 5011 cpu_manager.go:414] "containerMap: removing container" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.692252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692048 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.692252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692061 5011 cpu_manager.go:414] "containerMap: removing container" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.692252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692068 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.692447 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692421 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e580a1-6734-4810-aa37-d7db2739748a" containerName="revoke-keys" Oct 31 01:17:49.692447 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692438 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.692518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.692449 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dba605-8432-495d-a997-50c8b6338fa1" containerName="istio-proxy" Oct 31 01:17:49.698309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.698287 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.712429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.712387 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94"] Oct 31 01:17:49.712846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.712825 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:49.713479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.713431 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:17:49.790057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.790024 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.790270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.790246 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg88v\" (UniqueName: \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.790362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.790318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.891498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.891457 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg88v\" (UniqueName: \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.891699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.891552 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.891699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.891660 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.892436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.892407 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.894289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.894261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:49.914874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:49.914845 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg88v\" (UniqueName: \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\") pod \"maas-api-revoke-sub-33443aec-clm94\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:50.020155 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:50.019792 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:50.339841 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:17:50.339802 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fa412f_d98b_4bd6_ba9c_f53428348476.slice/crio-6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44 WatchSource:0}: Error finding container 6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44: Status 404 returned error can't find the container with id 6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44 Oct 31 01:17:50.341138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:50.341100 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94"] Oct 31 01:17:50.813803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:50.813769 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" event={"ID":"e4fa412f-d98b-4bd6-ba9c-f53428348476","Type":"ContainerStarted","Data":"39634968511c2996269feaab63b5e02c6dde77a37e24be5a24979bf4a306abce"} Oct 31 01:17:50.814196 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:50.813818 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" event={"ID":"e4fa412f-d98b-4bd6-ba9c-f53428348476","Type":"ContainerStarted","Data":"6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44"} Oct 31 01:17:50.856220 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:50.856156 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" podStartSLOduration=1.856141211 podStartE2EDuration="1.856141211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:50.853345773 +0000 UTC m=+3539.302433412" watchObservedRunningTime="2026-10-31 01:17:50.856141211 +0000 UTC m=+3539.305228850" Oct 31 01:17:51.823747 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:51.823707 5011 generic.go:356] "Generic (PLEG): container finished" podID="e4fa412f-d98b-4bd6-ba9c-f53428348476" containerID="39634968511c2996269feaab63b5e02c6dde77a37e24be5a24979bf4a306abce" exitCode=0 Oct 31 01:17:51.824134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:51.823786 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" event={"ID":"e4fa412f-d98b-4bd6-ba9c-f53428348476","Type":"ContainerDied","Data":"39634968511c2996269feaab63b5e02c6dde77a37e24be5a24979bf4a306abce"} Oct 31 01:17:53.004635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.004607 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:53.136330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.136258 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\" (UniqueName: \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\") pod \"e4fa412f-d98b-4bd6-ba9c-f53428348476\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " Oct 31 01:17:53.136330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.136324 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\") pod \"e4fa412f-d98b-4bd6-ba9c-f53428348476\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " Oct 31 01:17:53.136546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.136410 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\") pod \"e4fa412f-d98b-4bd6-ba9c-f53428348476\" (UID: \"e4fa412f-d98b-4bd6-ba9c-f53428348476\") " Oct 31 01:17:53.136663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.136639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca" pod "e4fa412f-d98b-4bd6-ba9c-f53428348476" (UID: "e4fa412f-d98b-4bd6-ba9c-f53428348476"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:53.136923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.136900 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:53.138553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.138517 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca" pod "e4fa412f-d98b-4bd6-ba9c-f53428348476" (UID: "e4fa412f-d98b-4bd6-ba9c-f53428348476"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:53.138948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.138927 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v" pod "e4fa412f-d98b-4bd6-ba9c-f53428348476" (UID: "e4fa412f-d98b-4bd6-ba9c-f53428348476"). InnerVolumeSpecName "kube-api-access-rg88v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:53.238624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.238588 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg88v\" (UniqueName: \"kubernetes.io/projected/e4fa412f-d98b-4bd6-ba9c-f53428348476-kube-api-access-rg88v\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:53.238624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.238620 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa412f-d98b-4bd6-ba9c-f53428348476-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:17:53.844188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.844153 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" Oct 31 01:17:53.844188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.844175 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94" event={"ID":"e4fa412f-d98b-4bd6-ba9c-f53428348476","Type":"ContainerDied","Data":"6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44"} Oct 31 01:17:53.844426 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:53.844242 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6446393f057e35f118b39bb91a86d5fb89e8c9d44be61ffbffcbaa4fbe858e44" Oct 31 01:17:54.769957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.769923 5011 scope.go:122] "RemoveContainer" containerID="8032363debe9aedafeca914f1f232ca8358dea6eba5cd69e958ef603ee66708d" Oct 31 01:17:54.781127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.781096 5011 scope.go:122] "RemoveContainer" containerID="aa443b70de06398c575c936d85451f89638e76be78f2159182d19a46b69b68e4" Oct 31 01:17:54.791376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.791355 5011 scope.go:122] "RemoveContainer" containerID="8a7f3a5c446589cf1a6ba75744e39c4dd782b497a8d5996479453d6389a28675" Oct 31 01:17:54.804695 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.804666 5011 scope.go:122] "RemoveContainer" containerID="4e8dbcff2ddbfb0c8769edcc965c9890ae064bda901e53e0f40d6d5a4d03907d" Oct 31 01:17:54.816404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.816379 5011 scope.go:122] "RemoveContainer" containerID="749b69b7a592aa77ee021cd3d3cd4c22902e1be72b28a77f3ff44263cca2e44d" Oct 31 01:17:54.828283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:54.828194 5011 scope.go:122] "RemoveContainer" containerID="c59397ebf2813776ab0072ebe9bbe876e7bca13a0f54d8b2329c2c5674aceed6" Oct 31 01:17:57.364737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.364695 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4"] Oct 31 01:17:57.365707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.365681 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e4fa412f-d98b-4bd6-ba9c-f53428348476" containerName="revoke-keys" Oct 31 01:17:57.365707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.365707 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fa412f-d98b-4bd6-ba9c-f53428348476" containerName="revoke-keys" Oct 31 01:17:57.365882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.365863 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fa412f-d98b-4bd6-ba9c-f53428348476" containerName="revoke-keys" Oct 31 01:17:57.369824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.369805 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.384512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.384487 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4"] Oct 31 01:17:57.391070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.391036 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:57.391070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.391059 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:17:57.449229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.448268 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.449229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.449004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkq8\" (UniqueName: \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.449229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.449080 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.550809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.550774 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.551020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.550858 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkq8\" (UniqueName: \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.551020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.550900 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.551635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.551613 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.553355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.553327 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.573930 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.573906 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkq8\" (UniqueName: \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\") pod \"maas-api-revoke-sub-59346ee6-vvsp4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:57.686425 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:57.686356 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:17:58.008497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:58.008467 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4"] Oct 31 01:17:58.009411 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:17:58.009382 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580ea84d_5430_4a21_9d81_8ada37729dc4.slice/crio-1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763 WatchSource:0}: Error finding container 1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763: Status 404 returned error can't find the container with id 1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763 Oct 31 01:17:58.909460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:58.909429 5011 generic.go:356] "Generic (PLEG): container finished" podID="580ea84d-5430-4a21-9d81-8ada37729dc4" containerID="7b47635b253b81211deb658e6cff36db7e971b3a17caf39140ed92efe1c2d150" exitCode=0 Oct 31 01:17:58.909827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:58.909476 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" event={"ID":"580ea84d-5430-4a21-9d81-8ada37729dc4","Type":"ContainerDied","Data":"7b47635b253b81211deb658e6cff36db7e971b3a17caf39140ed92efe1c2d150"} Oct 31 01:17:58.909827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:17:58.909516 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" event={"ID":"580ea84d-5430-4a21-9d81-8ada37729dc4","Type":"ContainerStarted","Data":"1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763"} Oct 31 01:18:00.085404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.085380 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:18:00.183531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.183443 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g"] Oct 31 01:18:00.184062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.184046 5011 cpu_manager.go:414] "containerMap: removing container" podUID="580ea84d-5430-4a21-9d81-8ada37729dc4" containerName="revoke-keys" Oct 31 01:18:00.184062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.184061 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580ea84d-5430-4a21-9d81-8ada37729dc4" containerName="revoke-keys" Oct 31 01:18:00.184229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.184202 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="580ea84d-5430-4a21-9d81-8ada37729dc4" containerName="revoke-keys" Oct 31 01:18:00.187609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.187585 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.187899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.187876 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\") pod \"580ea84d-5430-4a21-9d81-8ada37729dc4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " Oct 31 01:18:00.187977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.187927 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\") pod \"580ea84d-5430-4a21-9d81-8ada37729dc4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " Oct 31 01:18:00.188047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.187994 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\" (UniqueName: \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\") pod \"580ea84d-5430-4a21-9d81-8ada37729dc4\" (UID: \"580ea84d-5430-4a21-9d81-8ada37729dc4\") " Oct 31 01:18:00.188346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.188320 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca" pod "580ea84d-5430-4a21-9d81-8ada37729dc4" (UID: "580ea84d-5430-4a21-9d81-8ada37729dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:00.188717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.188694 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:00.190728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.190702 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8" pod "580ea84d-5430-4a21-9d81-8ada37729dc4" (UID: "580ea84d-5430-4a21-9d81-8ada37729dc4"). InnerVolumeSpecName "kube-api-access-pqkq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:00.190833 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.190752 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca" pod "580ea84d-5430-4a21-9d81-8ada37729dc4" (UID: "580ea84d-5430-4a21-9d81-8ada37729dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:00.205524 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.205497 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g"] Oct 31 01:18:00.289444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.289404 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.289634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.289520 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnjg\" (UniqueName: \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.289713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.289701 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.289782 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.289769 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580ea84d-5430-4a21-9d81-8ada37729dc4-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:00.289842 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.289824 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqkq8\" (UniqueName: \"kubernetes.io/projected/580ea84d-5430-4a21-9d81-8ada37729dc4-kube-api-access-pqkq8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:00.391312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.391277 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.391511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.391345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.391511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.391408 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnjg\" (UniqueName: \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.392069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.392047 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.393784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.393759 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.415088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.415060 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnjg\" (UniqueName: \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\") pod \"maas-api-revoke-sub-b727ecf5-rcw4g\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.533766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.533692 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:00.853407 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:00.853371 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61758a54_65dd_49e5_93c7_ff89f2f1b900.slice/crio-3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c WatchSource:0}: Error finding container 3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c: Status 404 returned error can't find the container with id 3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c Oct 31 01:18:00.853826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.853808 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g"] Oct 31 01:18:00.926005 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.925976 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" event={"ID":"61758a54-65dd-49e5-93c7-ff89f2f1b900","Type":"ContainerStarted","Data":"3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c"} Oct 31 01:18:00.927634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.927614 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" Oct 31 01:18:00.927759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.927619 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4" event={"ID":"580ea84d-5430-4a21-9d81-8ada37729dc4","Type":"ContainerDied","Data":"1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763"} Oct 31 01:18:00.927759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:00.927728 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fff56657e8655d8d6d5a831231f1291cdad75ed88526e7d8c8c8705a0bfc763" Oct 31 01:18:01.936852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:01.936820 5011 generic.go:356] "Generic (PLEG): container finished" podID="61758a54-65dd-49e5-93c7-ff89f2f1b900" containerID="afda80994ad5e8595d8fdd7c4151b59f2baa1476ce26b26990d0ce19d6625a2a" exitCode=0 Oct 31 01:18:01.937248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:01.936879 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" event={"ID":"61758a54-65dd-49e5-93c7-ff89f2f1b900","Type":"ContainerDied","Data":"afda80994ad5e8595d8fdd7c4151b59f2baa1476ce26b26990d0ce19d6625a2a"} Oct 31 01:18:01.948620 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:18:01.948585 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:18:03.137851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.137823 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:03.229642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.229552 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\" (UniqueName: \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\") pod \"61758a54-65dd-49e5-93c7-ff89f2f1b900\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " Oct 31 01:18:03.229817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.229646 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\") pod \"61758a54-65dd-49e5-93c7-ff89f2f1b900\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " Oct 31 01:18:03.229817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.229696 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\") pod \"61758a54-65dd-49e5-93c7-ff89f2f1b900\" (UID: \"61758a54-65dd-49e5-93c7-ff89f2f1b900\") " Oct 31 01:18:03.230178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.230147 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca" pod "61758a54-65dd-49e5-93c7-ff89f2f1b900" (UID: "61758a54-65dd-49e5-93c7-ff89f2f1b900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:03.230427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.230405 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:03.231947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.231903 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca" pod "61758a54-65dd-49e5-93c7-ff89f2f1b900" (UID: "61758a54-65dd-49e5-93c7-ff89f2f1b900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:03.232063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.231996 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg" pod "61758a54-65dd-49e5-93c7-ff89f2f1b900" (UID: "61758a54-65dd-49e5-93c7-ff89f2f1b900"). InnerVolumeSpecName "kube-api-access-6dnjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:03.332009 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.331977 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61758a54-65dd-49e5-93c7-ff89f2f1b900-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:03.332009 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.332011 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnjg\" (UniqueName: \"kubernetes.io/projected/61758a54-65dd-49e5-93c7-ff89f2f1b900-kube-api-access-6dnjg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:03.955929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.955853 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" Oct 31 01:18:03.955929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.955875 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g" event={"ID":"61758a54-65dd-49e5-93c7-ff89f2f1b900","Type":"ContainerDied","Data":"3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c"} Oct 31 01:18:03.956116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:03.955930 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db720e89f168dbac28c8b7541876320b3a08488ab22623b12a6c7eedc282c3c" Oct 31 01:18:07.695183 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.695150 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f"] Oct 31 01:18:07.695748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.695732 5011 cpu_manager.go:414] "containerMap: removing container" podUID="61758a54-65dd-49e5-93c7-ff89f2f1b900" containerName="revoke-keys" Oct 31 01:18:07.695810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.695750 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61758a54-65dd-49e5-93c7-ff89f2f1b900" containerName="revoke-keys" Oct 31 01:18:07.695878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.695869 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="61758a54-65dd-49e5-93c7-ff89f2f1b900" containerName="revoke-keys" Oct 31 01:18:07.699543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.699525 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.724800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.724758 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:07.724926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.724828 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:18:07.728527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.728501 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f"] Oct 31 01:18:07.792700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.792663 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.792855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.792843 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wmr\" (UniqueName: \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.793200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.793161 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.894310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.894279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wmr\" (UniqueName: \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.894459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.894442 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.894541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.894525 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.895156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.895134 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.912877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.897140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:07.920847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:07.920778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wmr\" (UniqueName: \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\") pod \"maas-api-revoke-sub-892ec1da-x4f4f\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:08.016022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:08.015943 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:08.338876 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:08.338845 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee39e65_a8cd_49ca_96f6_8ee31baaf43b.slice/crio-8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a WatchSource:0}: Error finding container 8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a: Status 404 returned error can't find the container with id 8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a Oct 31 01:18:08.341760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:08.341655 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f"] Oct 31 01:18:09.002448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:09.002410 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" event={"ID":"fee39e65-a8cd-49ca-96f6-8ee31baaf43b","Type":"ContainerStarted","Data":"8469c51a6baf7c2727d15657e21a1ef3695789ca383de617f5847972c75ea5b8"} Oct 31 01:18:09.002843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:09.002470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" event={"ID":"fee39e65-a8cd-49ca-96f6-8ee31baaf43b","Type":"ContainerStarted","Data":"8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a"} Oct 31 01:18:09.046619 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:09.046556 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" podStartSLOduration=2.0465409980000002 podStartE2EDuration="2.046540998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:09.042575779 +0000 UTC m=+3557.491663401" watchObservedRunningTime="2026-10-31 01:18:09.046540998 +0000 UTC m=+3557.495628672" Oct 31 01:18:10.012191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:10.012154 5011 generic.go:356] "Generic (PLEG): container finished" podID="fee39e65-a8cd-49ca-96f6-8ee31baaf43b" containerID="8469c51a6baf7c2727d15657e21a1ef3695789ca383de617f5847972c75ea5b8" exitCode=0 Oct 31 01:18:10.012637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:10.012250 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" event={"ID":"fee39e65-a8cd-49ca-96f6-8ee31baaf43b","Type":"ContainerDied","Data":"8469c51a6baf7c2727d15657e21a1ef3695789ca383de617f5847972c75ea5b8"} Oct 31 01:18:10.071027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:10.070990 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f"] Oct 31 01:18:10.088084 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:10.088053 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-33d44f-nvg2f"] Oct 31 01:18:11.111327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.111288 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj"] Oct 31 01:18:11.116723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.116698 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.132399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.132367 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj"] Oct 31 01:18:11.199901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.199872 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:11.248794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.248762 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\") pod \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " Oct 31 01:18:11.248960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.248944 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\") pod \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " Oct 31 01:18:11.249112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.249090 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\" (UniqueName: \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\") pod \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\" (UID: \"fee39e65-a8cd-49ca-96f6-8ee31baaf43b\") " Oct 31 01:18:11.249294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.249177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca" pod "fee39e65-a8cd-49ca-96f6-8ee31baaf43b" (UID: "fee39e65-a8cd-49ca-96f6-8ee31baaf43b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:11.249670 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.249635 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.249767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.249702 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgj6\" (UniqueName: \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.249936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.249919 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.250083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.250060 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:11.251454 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.251431 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca" pod "fee39e65-a8cd-49ca-96f6-8ee31baaf43b" (UID: "fee39e65-a8cd-49ca-96f6-8ee31baaf43b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:11.252382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.252353 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr" pod "fee39e65-a8cd-49ca-96f6-8ee31baaf43b" (UID: "fee39e65-a8cd-49ca-96f6-8ee31baaf43b"). InnerVolumeSpecName "kube-api-access-c9wmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:11.351304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.351268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.351485 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.351461 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.351647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.351494 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgj6\" (UniqueName: \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.351727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.351678 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wmr\" (UniqueName: \"kubernetes.io/projected/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-kube-api-access-c9wmr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:11.351727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.351701 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee39e65-a8cd-49ca-96f6-8ee31baaf43b-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:11.352164 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.352146 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.353907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.353886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.375563 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.375537 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgj6\" (UniqueName: \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\") pod \"maas-api-revoke-sub-a67d8597-6bjgj\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.458781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.458754 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:11.684758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.684681 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e580a1-6734-4810-aa37-d7db2739748a" path="/var/lib/kubelet/pods/72e580a1-6734-4810-aa37-d7db2739748a/volumes" Oct 31 01:18:11.765547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:11.765517 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj"] Oct 31 01:18:11.766158 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:11.766129 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881dfccb_42ae_4332_a940_c289aa970527.slice/crio-f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a WatchSource:0}: Error finding container f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a: Status 404 returned error can't find the container with id f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a Oct 31 01:18:12.030537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:12.030396 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" Oct 31 01:18:12.030537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:12.030422 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f" event={"ID":"fee39e65-a8cd-49ca-96f6-8ee31baaf43b","Type":"ContainerDied","Data":"8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a"} Oct 31 01:18:12.030537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:12.030475 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8288654fda48ec69443a0345111fae120f983f091d8d407120f12764557bb24a" Oct 31 01:18:12.032016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:12.031984 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" event={"ID":"881dfccb-42ae-4332-a940-c289aa970527","Type":"ContainerStarted","Data":"f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a"} Oct 31 01:18:13.043495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.043405 5011 generic.go:356] "Generic (PLEG): container finished" podID="881dfccb-42ae-4332-a940-c289aa970527" containerID="2a2b88e877e811705bdd1c0520a1d5111dd31a9cf67f6f381b3c430f0189f3d2" exitCode=0 Oct 31 01:18:13.043863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.043489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" event={"ID":"881dfccb-42ae-4332-a940-c289aa970527","Type":"ContainerDied","Data":"2a2b88e877e811705bdd1c0520a1d5111dd31a9cf67f6f381b3c430f0189f3d2"} Oct 31 01:18:13.621865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.621825 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:18:13.622793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.622765 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fee39e65-a8cd-49ca-96f6-8ee31baaf43b" containerName="revoke-keys" Oct 31 01:18:13.622793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.622792 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee39e65-a8cd-49ca-96f6-8ee31baaf43b" containerName="revoke-keys" Oct 31 01:18:13.623041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.623027 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee39e65-a8cd-49ca-96f6-8ee31baaf43b" containerName="revoke-keys" Oct 31 01:18:13.627841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.627817 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.652874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.649892 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:18:13.657034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.655907 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4207cd\"/\"ipp-route-4207cd-kserve-self-signed-certs\"" Oct 31 01:18:13.657034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.656033 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4207cd\"/\"default-dockercfg-2gmwx\"" Oct 31 01:18:13.657034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.657018 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4207cd\"/\"kube-root-ca.crt\"" Oct 31 01:18:13.657344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.657157 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4207cd\"/\"openshift-service-ca.crt\"" Oct 31 01:18:13.687609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.687575 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.688218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.688084 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.688393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.688242 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.688495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.688414 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.688599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.688522 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.688677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.688635 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn9k\" (UniqueName: \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790001 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790064 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790079 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790130 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790189 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790295 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn9k\" (UniqueName: \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790537 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.790913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.790881 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.792650 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.792628 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.792974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.792954 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.824884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.824856 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn9k\" (UniqueName: \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\") pod \"ipp-route-4207cd-kserve-6b48c4cb45-7w4rb\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:13.972277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:13.970245 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:14.289029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.288997 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:14.320931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.320899 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:18:14.321061 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:14.320975 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd72c884c_40e1_45d5_bb7a_6cf677d95b8e.slice/crio-fe27477788d432be4eec0af5705b67eedab1c3746074ad159f640e87e7652012 WatchSource:0}: Error finding container fe27477788d432be4eec0af5705b67eedab1c3746074ad159f640e87e7652012: Status 404 returned error can't find the container with id fe27477788d432be4eec0af5705b67eedab1c3746074ad159f640e87e7652012 Oct 31 01:18:14.414472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.414443 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\" (UniqueName: \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\") pod \"881dfccb-42ae-4332-a940-c289aa970527\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " Oct 31 01:18:14.414635 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.414594 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\") pod \"881dfccb-42ae-4332-a940-c289aa970527\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " Oct 31 01:18:14.414750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.414736 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\") pod \"881dfccb-42ae-4332-a940-c289aa970527\" (UID: \"881dfccb-42ae-4332-a940-c289aa970527\") " Oct 31 01:18:14.415185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.414922 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca" pod "881dfccb-42ae-4332-a940-c289aa970527" (UID: "881dfccb-42ae-4332-a940-c289aa970527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:14.415341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.415289 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881dfccb-42ae-4332-a940-c289aa970527-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:14.417037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.416766 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6" pod "881dfccb-42ae-4332-a940-c289aa970527" (UID: "881dfccb-42ae-4332-a940-c289aa970527"). InnerVolumeSpecName "kube-api-access-ftgj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:14.417352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.417331 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca" pod "881dfccb-42ae-4332-a940-c289aa970527" (UID: "881dfccb-42ae-4332-a940-c289aa970527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:14.516736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.516699 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881dfccb-42ae-4332-a940-c289aa970527-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:14.516736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:14.516734 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgj6\" (UniqueName: \"kubernetes.io/projected/881dfccb-42ae-4332-a940-c289aa970527-kube-api-access-ftgj6\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:15.068178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.068139 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" event={"ID":"d72c884c-40e1-45d5-bb7a-6cf677d95b8e","Type":"ContainerStarted","Data":"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de"} Oct 31 01:18:15.068410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.068192 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" event={"ID":"d72c884c-40e1-45d5-bb7a-6cf677d95b8e","Type":"ContainerStarted","Data":"fe27477788d432be4eec0af5705b67eedab1c3746074ad159f640e87e7652012"} Oct 31 01:18:15.069639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.069611 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" Oct 31 01:18:15.069748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.069639 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj" event={"ID":"881dfccb-42ae-4332-a940-c289aa970527","Type":"ContainerDied","Data":"f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a"} Oct 31 01:18:15.069748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.069677 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ad4aa8460006584fbe3d90cfe3c5f910876758e9d3418daefa56d1db16be4a" Oct 31 01:18:15.113673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.113619 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" podStartSLOduration=2.113602286 podStartE2EDuration="2.113602286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:15.109105883 +0000 UTC m=+3563.558193524" watchObservedRunningTime="2026-10-31 01:18:15.113602286 +0000 UTC m=+3563.562689927" Oct 31 01:18:15.972113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:15.972083 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:17.880118 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.880079 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782"] Oct 31 01:18:17.880826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.880805 5011 cpu_manager.go:414] "containerMap: removing container" podUID="881dfccb-42ae-4332-a940-c289aa970527" containerName="revoke-keys" Oct 31 01:18:17.880826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.880826 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881dfccb-42ae-4332-a940-c289aa970527" containerName="revoke-keys" Oct 31 01:18:17.881060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.880938 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="881dfccb-42ae-4332-a940-c289aa970527" containerName="revoke-keys" Oct 31 01:18:17.900699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.900664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:17.906796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.906764 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782"] Oct 31 01:18:17.927484 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.927460 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:17.927712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:17.927686 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:18:18.071723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.071686 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2m44\" (UniqueName: \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.071919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.071878 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.071980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.071915 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.174459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.174374 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.174459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.174413 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.174720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.174575 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2m44\" (UniqueName: \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.175202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.175178 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.177150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.177126 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.197610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.197583 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2m44\" (UniqueName: \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.217906 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.217878 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:18.538087 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:18.538039 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0c953a_5b72_4b9a_9c09_b4e9508b6dc3.slice/crio-2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0 WatchSource:0}: Error finding container 2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0: Status 404 returned error can't find the container with id 2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0 Oct 31 01:18:18.539510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.539480 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782"] Oct 31 01:18:18.709154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.709121 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq"] Oct 31 01:18:18.714920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.714898 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.729152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.729129 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq"] Oct 31 01:18:18.781312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.781280 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.781462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.781429 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4h2\" (UniqueName: \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.781514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.781476 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.883108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.883071 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.883572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.883238 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4h2\" (UniqueName: \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.883572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.883329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.884117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.884091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.885563 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.885545 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:18.908991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:18.908970 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4h2\" (UniqueName: \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\") pod \"maas-api-revoke-sub-e5597e43-ckmbq\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:19.030789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:19.030762 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:19.128179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:19.128133 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" event={"ID":"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3","Type":"ContainerStarted","Data":"7607ce3f66e43e635eab3aa635d09254bcafacfc78b882835ec2c4a653b4473d"} Oct 31 01:18:19.128341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:19.128234 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" event={"ID":"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3","Type":"ContainerStarted","Data":"2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0"} Oct 31 01:18:19.175432 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:19.175381 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" podStartSLOduration=2.17536402 podStartE2EDuration="2.17536402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:19.169780505 +0000 UTC m=+3567.618868145" watchObservedRunningTime="2026-10-31 01:18:19.17536402 +0000 UTC m=+3567.624451662" Oct 31 01:18:19.345954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:19.345910 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq"] Oct 31 01:18:19.347478 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:19.347414 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0802494_f13e_41f6_adbc_ca03e491f689.slice/crio-793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39 WatchSource:0}: Error finding container 793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39: Status 404 returned error can't find the container with id 793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39 Oct 31 01:18:20.138928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:20.138886 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" containerID="7607ce3f66e43e635eab3aa635d09254bcafacfc78b882835ec2c4a653b4473d" exitCode=0 Oct 31 01:18:20.139403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:20.138966 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" event={"ID":"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3","Type":"ContainerDied","Data":"7607ce3f66e43e635eab3aa635d09254bcafacfc78b882835ec2c4a653b4473d"} Oct 31 01:18:20.141055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:20.141030 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" event={"ID":"a0802494-f13e-41f6-adbc-ca03e491f689","Type":"ContainerStarted","Data":"47414a9fa072ac6bbf82bac61aadb931f76e2558cccf48139c52e97bcdc41999"} Oct 31 01:18:20.141233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:20.141077 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" event={"ID":"a0802494-f13e-41f6-adbc-ca03e491f689","Type":"ContainerStarted","Data":"793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39"} Oct 31 01:18:20.226770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:20.226708 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" podStartSLOduration=2.226690539 podStartE2EDuration="2.226690539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:20.221435542 +0000 UTC m=+3568.670523181" watchObservedRunningTime="2026-10-31 01:18:20.226690539 +0000 UTC m=+3568.675778180" Oct 31 01:18:21.151891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.151854 5011 generic.go:356] "Generic (PLEG): container finished" podID="a0802494-f13e-41f6-adbc-ca03e491f689" containerID="47414a9fa072ac6bbf82bac61aadb931f76e2558cccf48139c52e97bcdc41999" exitCode=0 Oct 31 01:18:21.152404 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.151929 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" event={"ID":"a0802494-f13e-41f6-adbc-ca03e491f689","Type":"ContainerDied","Data":"47414a9fa072ac6bbf82bac61aadb931f76e2558cccf48139c52e97bcdc41999"} Oct 31 01:18:21.339877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.339847 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:21.416590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.416504 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\") pod \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " Oct 31 01:18:21.416764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.416685 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\") pod \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " Oct 31 01:18:21.416764 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.416725 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\" (UniqueName: \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\") pod \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\" (UID: \"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3\") " Oct 31 01:18:21.416959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.416930 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca" pod "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" (UID: "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:21.417371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.417344 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:21.419468 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.419444 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44" pod "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" (UID: "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3"). InnerVolumeSpecName "kube-api-access-m2m44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:21.420115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.420087 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca" pod "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" (UID: "0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:21.533090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.533046 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:21.533090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.533090 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2m44\" (UniqueName: \"kubernetes.io/projected/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3-kube-api-access-m2m44\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:21.722975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.722943 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk"] Oct 31 01:18:21.723566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.723552 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" containerName="revoke-keys" Oct 31 01:18:21.723619 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.723568 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" containerName="revoke-keys" Oct 31 01:18:21.723669 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.723660 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" containerName="revoke-keys" Oct 31 01:18:21.727236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.727220 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.745136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.745112 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk"] Oct 31 01:18:21.836957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.836920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmq5\" (UniqueName: \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.837150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.836984 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.837150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.837134 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.938549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.938514 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmq5\" (UniqueName: \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.938760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.938584 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.938760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.938651 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.939379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.939352 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.941587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.941559 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:21.961848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:21.961791 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmq5\" (UniqueName: \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\") pod \"maas-api-revoke-sub-73fdcea1-8trwk\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:22.046327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.046256 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:22.171890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.171857 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" Oct 31 01:18:22.172498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.172469 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782" event={"ID":"0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3","Type":"ContainerDied","Data":"2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0"} Oct 31 01:18:22.176223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.175956 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a17642cf3c1d74dfb868e4567302c5a8057c673ff6db591f36514f474ce72c0" Oct 31 01:18:22.366551 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.366510 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:22.368874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.368847 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk"] Oct 31 01:18:22.369327 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:22.369300 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0424739b_f68f_4ebf_80f7_9f6cb7f621f2.slice/crio-823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0 WatchSource:0}: Error finding container 823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0: Status 404 returned error can't find the container with id 823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0 Oct 31 01:18:22.464226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.460157 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\" (UniqueName: \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\") pod \"a0802494-f13e-41f6-adbc-ca03e491f689\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " Oct 31 01:18:22.464226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.460265 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\") pod \"a0802494-f13e-41f6-adbc-ca03e491f689\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " Oct 31 01:18:22.464226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.460554 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\") pod \"a0802494-f13e-41f6-adbc-ca03e491f689\" (UID: \"a0802494-f13e-41f6-adbc-ca03e491f689\") " Oct 31 01:18:22.464226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.462311 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca" pod "a0802494-f13e-41f6-adbc-ca03e491f689" (UID: "a0802494-f13e-41f6-adbc-ca03e491f689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:22.518865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.518759 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca" pod "a0802494-f13e-41f6-adbc-ca03e491f689" (UID: "a0802494-f13e-41f6-adbc-ca03e491f689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:22.518865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.518801 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2" pod "a0802494-f13e-41f6-adbc-ca03e491f689" (UID: "a0802494-f13e-41f6-adbc-ca03e491f689"). InnerVolumeSpecName "kube-api-access-8d4h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:22.562082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.562049 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:22.562082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.562078 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4h2\" (UniqueName: \"kubernetes.io/projected/a0802494-f13e-41f6-adbc-ca03e491f689-kube-api-access-8d4h2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:22.562082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:22.562088 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0802494-f13e-41f6-adbc-ca03e491f689-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:23.182046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.181958 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" event={"ID":"0424739b-f68f-4ebf-80f7-9f6cb7f621f2","Type":"ContainerStarted","Data":"872fc8d736e661a61fa7ab4e5a125c5955df5df79046c7eb7ea9eeae773a071a"} Oct 31 01:18:23.182046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.182021 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" event={"ID":"0424739b-f68f-4ebf-80f7-9f6cb7f621f2","Type":"ContainerStarted","Data":"823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0"} Oct 31 01:18:23.183520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.183490 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" Oct 31 01:18:23.183520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.183506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq" event={"ID":"a0802494-f13e-41f6-adbc-ca03e491f689","Type":"ContainerDied","Data":"793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39"} Oct 31 01:18:23.183682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.183546 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793f69fa231762c3d17cc0d3a8be7c4881821de02b661c48213f67b8bc85cc39" Oct 31 01:18:23.230984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.230936 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" podStartSLOduration=2.230922985 podStartE2EDuration="2.230922985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:23.229455809 +0000 UTC m=+3571.678543447" watchObservedRunningTime="2026-10-31 01:18:23.230922985 +0000 UTC m=+3571.680010660" Oct 31 01:18:23.971606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.971550 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:23.984976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:23.984946 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:24.193388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:24.193353 5011 generic.go:356] "Generic (PLEG): container finished" podID="0424739b-f68f-4ebf-80f7-9f6cb7f621f2" containerID="872fc8d736e661a61fa7ab4e5a125c5955df5df79046c7eb7ea9eeae773a071a" exitCode=0 Oct 31 01:18:24.193809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:24.193436 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" event={"ID":"0424739b-f68f-4ebf-80f7-9f6cb7f621f2","Type":"ContainerDied","Data":"872fc8d736e661a61fa7ab4e5a125c5955df5df79046c7eb7ea9eeae773a071a"} Oct 31 01:18:24.204750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:24.204724 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:18:25.378992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.378966 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:25.401823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.401798 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\") pod \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " Oct 31 01:18:25.401987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.401872 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\") pod \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " Oct 31 01:18:25.402193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.402171 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca" pod "0424739b-f68f-4ebf-80f7-9f6cb7f621f2" (UID: "0424739b-f68f-4ebf-80f7-9f6cb7f621f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:25.402294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.402228 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\" (UniqueName: \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\") pod \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\" (UID: \"0424739b-f68f-4ebf-80f7-9f6cb7f621f2\") " Oct 31 01:18:25.402822 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.402794 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:25.404561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.404537 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca" pod "0424739b-f68f-4ebf-80f7-9f6cb7f621f2" (UID: "0424739b-f68f-4ebf-80f7-9f6cb7f621f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:25.404710 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.404683 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5" pod "0424739b-f68f-4ebf-80f7-9f6cb7f621f2" (UID: "0424739b-f68f-4ebf-80f7-9f6cb7f621f2"). InnerVolumeSpecName "kube-api-access-qgmq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:25.504548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.504474 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmq5\" (UniqueName: \"kubernetes.io/projected/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-kube-api-access-qgmq5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:25.504548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:25.504505 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0424739b-f68f-4ebf-80f7-9f6cb7f621f2-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:26.212832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:26.212801 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" Oct 31 01:18:26.213008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:26.212832 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk" event={"ID":"0424739b-f68f-4ebf-80f7-9f6cb7f621f2","Type":"ContainerDied","Data":"823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0"} Oct 31 01:18:26.213008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:26.212888 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823bcc54bb406bea0a2c91da301e1ea9939f8328d41d46a6368f77b15c1d7fb0" Oct 31 01:18:29.394228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.393826 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb"] Oct 31 01:18:29.394876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.394855 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a0802494-f13e-41f6-adbc-ca03e491f689" containerName="revoke-keys" Oct 31 01:18:29.394955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.394880 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0802494-f13e-41f6-adbc-ca03e491f689" containerName="revoke-keys" Oct 31 01:18:29.394955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.394899 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0424739b-f68f-4ebf-80f7-9f6cb7f621f2" containerName="revoke-keys" Oct 31 01:18:29.394955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.394909 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0424739b-f68f-4ebf-80f7-9f6cb7f621f2" containerName="revoke-keys" Oct 31 01:18:29.395123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.395081 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0802494-f13e-41f6-adbc-ca03e491f689" containerName="revoke-keys" Oct 31 01:18:29.395123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.395097 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0424739b-f68f-4ebf-80f7-9f6cb7f621f2" containerName="revoke-keys" Oct 31 01:18:29.434232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.407594 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.434232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.423279 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb"] Oct 31 01:18:29.434232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.423850 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:29.437244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.435963 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:18:29.575348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.575310 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.575559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.575453 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.575646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.575624 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwczn\" (UniqueName: \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.678080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.677980 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.678283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.678113 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.678283 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.678265 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwczn\" (UniqueName: \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.679042 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.679016 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.693251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.681899 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.722077 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.709787 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwczn\" (UniqueName: \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\") pod \"maas-api-revoke-sub-5b0432b9-kzmcb\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:29.731760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:29.731710 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:30.116761 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:18:30.116729 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d47532_dbb5_4b7e_8934_a96fd7043ae3.slice/crio-3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae WatchSource:0}: Error finding container 3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae: Status 404 returned error can't find the container with id 3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae Oct 31 01:18:30.117061 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:30.116970 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb"] Oct 31 01:18:30.250067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:30.250037 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" event={"ID":"23d47532-dbb5-4b7e-8934-a96fd7043ae3","Type":"ContainerStarted","Data":"3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae"} Oct 31 01:18:31.260446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:31.260363 5011 generic.go:356] "Generic (PLEG): container finished" podID="23d47532-dbb5-4b7e-8934-a96fd7043ae3" containerID="b437bdba9939bce2618a84f7f196457b21e579949f62d36db0fee175ceac208f" exitCode=0 Oct 31 01:18:31.260446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:31.260427 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" event={"ID":"23d47532-dbb5-4b7e-8934-a96fd7043ae3","Type":"ContainerDied","Data":"b437bdba9939bce2618a84f7f196457b21e579949f62d36db0fee175ceac208f"} Oct 31 01:18:31.947909 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:18:31.947867 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:18:32.448807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.448780 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:32.518791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.518713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\" (UniqueName: \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\") pod \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " Oct 31 01:18:32.518948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.518810 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\") pod \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " Oct 31 01:18:32.519021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.518949 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\") pod \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\" (UID: \"23d47532-dbb5-4b7e-8934-a96fd7043ae3\") " Oct 31 01:18:32.520153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.520099 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca" pod "23d47532-dbb5-4b7e-8934-a96fd7043ae3" (UID: "23d47532-dbb5-4b7e-8934-a96fd7043ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:32.521673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.521643 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca" pod "23d47532-dbb5-4b7e-8934-a96fd7043ae3" (UID: "23d47532-dbb5-4b7e-8934-a96fd7043ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:32.521940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.521901 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn" pod "23d47532-dbb5-4b7e-8934-a96fd7043ae3" (UID: "23d47532-dbb5-4b7e-8934-a96fd7043ae3"). InnerVolumeSpecName "kube-api-access-bwczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:32.620856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.620818 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwczn\" (UniqueName: \"kubernetes.io/projected/23d47532-dbb5-4b7e-8934-a96fd7043ae3-kube-api-access-bwczn\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:32.620856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.620856 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:32.621096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:32.620872 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d47532-dbb5-4b7e-8934-a96fd7043ae3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:33.276831 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:33.276799 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" Oct 31 01:18:33.277030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:33.276830 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb" event={"ID":"23d47532-dbb5-4b7e-8934-a96fd7043ae3","Type":"ContainerDied","Data":"3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae"} Oct 31 01:18:33.277030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:33.276880 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e44b96268800c68aa6fc3934e897fafe86214d0da6892e912bcce3ecce9dcae" Oct 31 01:18:41.055468 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:41.055430 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:18:42.945022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:42.944974 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:18:46.143352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:46.135004 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:18:52.682370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:52.682342 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:18:52.704332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:52.704303 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:18:54.938936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:54.938901 5011 scope.go:122] "RemoveContainer" containerID="b13b57ab775f824f884f412357de143f83c6b5ace799513b24902df435ebb7f8" Oct 31 01:18:54.949980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:54.949960 5011 scope.go:122] "RemoveContainer" containerID="a02390dfccc506eff08d5d9a7b0bf02cf487c4cf16594c1e0144504474908bcf" Oct 31 01:18:56.036249 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.036200 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm"] Oct 31 01:18:56.048232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.048177 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89d45cc-qq7xm"] Oct 31 01:18:56.079530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.079498 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" podUID="8969133a-2929-4873-9721-f7bae064c00d" containerName="main" containerID="cri-o://5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179" gracePeriod=45 Oct 31 01:18:56.357414 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.357382 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:18:56.400540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400509 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400558 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400599 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400718 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400810 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\" (UniqueName: \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400890 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\") pod \"8969133a-2929-4873-9721-f7bae064c00d\" (UID: \"8969133a-2929-4873-9721-f7bae064c00d\") " Oct 31 01:18:56.400996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.400931 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:56.401324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.401281 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:56.401324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.401299 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:56.401567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.401516 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.401567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.401536 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.401567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.401550 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.404941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.404917 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:56.405057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.404972 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:56.408727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.408692 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz" pod "8969133a-2929-4873-9721-f7bae064c00d" (UID: "8969133a-2929-4873-9721-f7bae064c00d"). InnerVolumeSpecName "kube-api-access-ctzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:56.491354 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.491314 5011 generic.go:356] "Generic (PLEG): container finished" podID="8969133a-2929-4873-9721-f7bae064c00d" containerID="5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179" exitCode=0 Oct 31 01:18:56.491530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.491375 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" Oct 31 01:18:56.491530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.491394 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" event={"ID":"8969133a-2929-4873-9721-f7bae064c00d","Type":"ContainerDied","Data":"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179"} Oct 31 01:18:56.491530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.491449 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds" event={"ID":"8969133a-2929-4873-9721-f7bae064c00d","Type":"ContainerDied","Data":"e9ff2379c64c0681bda824bbff7654938828bb1f4a91ac3a5dba61c9a77e7a5d"} Oct 31 01:18:56.491530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.491474 5011 scope.go:122] "RemoveContainer" containerID="5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179" Oct 31 01:18:56.503370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.503349 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8969133a-2929-4873-9721-f7bae064c00d-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.503370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.503371 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8969133a-2929-4873-9721-f7bae064c00d-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.503516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.503380 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzrz\" (UniqueName: \"kubernetes.io/projected/8969133a-2929-4873-9721-f7bae064c00d-kube-api-access-ctzrz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:56.504729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.504710 5011 scope.go:122] "RemoveContainer" containerID="5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179" Oct 31 01:18:56.505019 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:18:56.504999 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179\": container with ID starting with 5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179 not found: ID does not exist" containerID="5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179" Oct 31 01:18:56.505082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.505027 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179"} err="failed to get container status \"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179\": rpc error: code = NotFound desc = could not find container \"5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179\": container with ID starting with 5def744d0cfc528e7206641f70828c89b69db0876617338ffd1cd16de4d7b179 not found: ID does not exist" Oct 31 01:18:56.568858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.568830 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:18:56.589344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:56.589268 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-36af31/sub-test-model-36af31-kserve-65c9674dd5-mvmds"] Oct 31 01:18:57.684789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:57.684754 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8969133a-2929-4873-9721-f7bae064c00d" path="/var/lib/kubelet/pods/8969133a-2929-4873-9721-f7bae064c00d/volumes" Oct 31 01:18:57.685318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:57.685299 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bb5a7d-53a0-488e-8dfa-e76a57b94c46" path="/var/lib/kubelet/pods/99bb5a7d-53a0-488e-8dfa-e76a57b94c46/volumes" Oct 31 01:18:57.970498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:57.970401 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" podUID="13342db5-6125-459d-a545-346edfb59fda" containerName="main" containerID="cri-o://275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411" gracePeriod=45 Oct 31 01:18:58.266365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.266332 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:18:58.329113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329078 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\" (UniqueName: \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.329304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329173 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.329304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329273 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.329488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329465 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:58.329553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329514 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.329612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.329668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.329632 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\" (UniqueName: \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\") pod \"13342db5-6125-459d-a545-346edfb59fda\" (UID: \"13342db5-6125-459d-a545-346edfb59fda\") " Oct 31 01:18:58.330377 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.330348 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.331004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.330957 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:58.331004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.330969 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:58.331844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.331822 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:58.332051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.332027 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "kube-api-access-64thh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:58.333433 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.333402 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs" pod "13342db5-6125-459d-a545-346edfb59fda" (UID: "13342db5-6125-459d-a545-346edfb59fda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:58.431420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.431381 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.431420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.431415 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.431420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.431424 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13342db5-6125-459d-a545-346edfb59fda-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.431736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.431434 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64thh\" (UniqueName: \"kubernetes.io/projected/13342db5-6125-459d-a545-346edfb59fda-kube-api-access-64thh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.431736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.431444 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13342db5-6125-459d-a545-346edfb59fda-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:18:58.511076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.510981 5011 generic.go:356] "Generic (PLEG): container finished" podID="13342db5-6125-459d-a545-346edfb59fda" containerID="275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411" exitCode=0 Oct 31 01:18:58.511076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.511028 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" Oct 31 01:18:58.511357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.511082 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" event={"ID":"13342db5-6125-459d-a545-346edfb59fda","Type":"ContainerDied","Data":"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411"} Oct 31 01:18:58.511357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.511124 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z" event={"ID":"13342db5-6125-459d-a545-346edfb59fda","Type":"ContainerDied","Data":"ba0b100e9811d13a5a6a3cc5f89e89d731bba77bf438c5019dc352065dabcf1d"} Oct 31 01:18:58.511357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.511143 5011 scope.go:122] "RemoveContainer" containerID="275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411" Oct 31 01:18:58.524568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.524543 5011 scope.go:122] "RemoveContainer" containerID="275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411" Oct 31 01:18:58.524826 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:18:58.524801 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411\": container with ID starting with 275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411 not found: ID does not exist" containerID="275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411" Oct 31 01:18:58.524919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.524833 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411"} err="failed to get container status \"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411\": rpc error: code = NotFound desc = could not find container \"275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411\": container with ID starting with 275dc189cc8c85c38b976f9fbca89eac9b07090bac3d5dd755973f0c32f0a411 not found: ID does not exist" Oct 31 01:18:58.568350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.568304 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:18:58.574919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:58.574888 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-498580/sub-test-model-498580-kserve-656c554fb9-kwt4z"] Oct 31 01:18:59.684563 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:18:59.684528 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13342db5-6125-459d-a545-346edfb59fda" path="/var/lib/kubelet/pods/13342db5-6125-459d-a545-346edfb59fda/volumes" Oct 31 01:19:01.167226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.166898 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" podUID="b65fd2cd-2192-4325-8e12-d106402bfe7e" containerName="main" containerID="cri-o://1f2a7cdfdecf4609a88b14e8b784ed2c29308eaf1749db281963cb035ce938db" gracePeriod=45 Oct 31 01:19:01.312489 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.312452 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:19:01.557042 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.556956 5011 generic.go:356] "Generic (PLEG): container finished" podID="b65fd2cd-2192-4325-8e12-d106402bfe7e" containerID="1f2a7cdfdecf4609a88b14e8b784ed2c29308eaf1749db281963cb035ce938db" exitCode=0 Oct 31 01:19:01.557245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.557126 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" event={"ID":"b65fd2cd-2192-4325-8e12-d106402bfe7e","Type":"ContainerDied","Data":"1f2a7cdfdecf4609a88b14e8b784ed2c29308eaf1749db281963cb035ce938db"} Oct 31 01:19:01.605253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.605200 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.677467 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\" (UniqueName: \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.677625 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.677722 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.677882 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.677948 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.679768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.678030 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\") pod \"b65fd2cd-2192-4325-8e12-d106402bfe7e\" (UID: \"b65fd2cd-2192-4325-8e12-d106402bfe7e\") " Oct 31 01:19:01.680599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.680556 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:01.681280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.680966 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:01.681280 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.681084 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:01.683194 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.682665 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:01.684322 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.684288 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:01.684460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.684316 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59" pod "b65fd2cd-2192-4325-8e12-d106402bfe7e" (UID: "b65fd2cd-2192-4325-8e12-d106402bfe7e"). InnerVolumeSpecName "kube-api-access-gnv59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:01.780136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780094 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnv59\" (UniqueName: \"kubernetes.io/projected/b65fd2cd-2192-4325-8e12-d106402bfe7e-kube-api-access-gnv59\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.780136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780132 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.780368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780150 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65fd2cd-2192-4325-8e12-d106402bfe7e-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.780368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780166 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.780368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780179 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.780368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.780193 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65fd2cd-2192-4325-8e12-d106402bfe7e-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:01.925270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.925234 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t"] Oct 31 01:19:01.926132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926110 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b65fd2cd-2192-4325-8e12-d106402bfe7e" containerName="main" Oct 31 01:19:01.926132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926132 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65fd2cd-2192-4325-8e12-d106402bfe7e" containerName="main" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926174 5011 cpu_manager.go:414] "containerMap: removing container" podUID="13342db5-6125-459d-a545-346edfb59fda" containerName="main" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926182 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13342db5-6125-459d-a545-346edfb59fda" containerName="main" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926192 5011 cpu_manager.go:414] "containerMap: removing container" podUID="23d47532-dbb5-4b7e-8934-a96fd7043ae3" containerName="revoke-keys" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926200 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d47532-dbb5-4b7e-8934-a96fd7043ae3" containerName="revoke-keys" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926235 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8969133a-2929-4873-9721-f7bae064c00d" containerName="main" Oct 31 01:19:01.926307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926243 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8969133a-2929-4873-9721-f7bae064c00d" containerName="main" Oct 31 01:19:01.926521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926413 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65fd2cd-2192-4325-8e12-d106402bfe7e" containerName="main" Oct 31 01:19:01.926521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926432 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="13342db5-6125-459d-a545-346edfb59fda" containerName="main" Oct 31 01:19:01.926521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926441 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d47532-dbb5-4b7e-8934-a96fd7043ae3" containerName="revoke-keys" Oct 31 01:19:01.926521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.926450 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8969133a-2929-4873-9721-f7bae064c00d" containerName="main" Oct 31 01:19:01.930746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.930720 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:01.946956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.946898 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t"] Oct 31 01:19:01.947983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.947957 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:01.948129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.948051 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:19:01.963253 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:19:01.962632 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:19:01.982966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.982911 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:01.983160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.983139 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:01.983361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:01.983340 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsh9\" (UniqueName: \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.084543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.084507 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.084715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.084626 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsh9\" (UniqueName: \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.084792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.084760 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.085438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.085412 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.087057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.087035 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.108483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.108458 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsh9\" (UniqueName: \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.249925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.249844 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:02.560256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.560221 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t"] Oct 31 01:19:02.561058 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:19:02.561033 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3448ba_cd49_4b20_ae59_d1c0cb905d70.slice/crio-dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781 WatchSource:0}: Error finding container dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781: Status 404 returned error can't find the container with id dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781 Oct 31 01:19:02.570297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.570256 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" event={"ID":"b65fd2cd-2192-4325-8e12-d106402bfe7e","Type":"ContainerDied","Data":"4e859003568ee780522bb05d36ac5a3af17876afaded54761c0c91ac5bdd0a73"} Oct 31 01:19:02.570297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.570287 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6" Oct 31 01:19:02.570479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.570313 5011 scope.go:122] "RemoveContainer" containerID="1f2a7cdfdecf4609a88b14e8b784ed2c29308eaf1749db281963cb035ce938db" Oct 31 01:19:02.624598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.624565 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:19:02.628639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:02.628611 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f2bdf3/sub-test-model-f2bdf3-kserve-747bb97879-c55v6"] Oct 31 01:19:03.591169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:03.591128 5011 generic.go:356] "Generic (PLEG): container finished" podID="2d3448ba-cd49-4b20-ae59-d1c0cb905d70" containerID="c6e39777c82f097c703e3a1d6adee4724101b666f0702a9a301ded6d8e946edb" exitCode=0 Oct 31 01:19:03.591636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:03.591192 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" event={"ID":"2d3448ba-cd49-4b20-ae59-d1c0cb905d70","Type":"ContainerDied","Data":"c6e39777c82f097c703e3a1d6adee4724101b666f0702a9a301ded6d8e946edb"} Oct 31 01:19:03.591636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:03.591306 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" event={"ID":"2d3448ba-cd49-4b20-ae59-d1c0cb905d70","Type":"ContainerStarted","Data":"dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781"} Oct 31 01:19:03.705279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:03.701346 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65fd2cd-2192-4325-8e12-d106402bfe7e" path="/var/lib/kubelet/pods/b65fd2cd-2192-4325-8e12-d106402bfe7e/volumes" Oct 31 01:19:04.776241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.776197 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:04.817967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.817939 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\") pod \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " Oct 31 01:19:04.818160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.818034 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\") pod \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " Oct 31 01:19:04.818278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.818192 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\" (UniqueName: \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\") pod \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\" (UID: \"2d3448ba-cd49-4b20-ae59-d1c0cb905d70\") " Oct 31 01:19:04.819741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.818864 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca" pod "2d3448ba-cd49-4b20-ae59-d1c0cb905d70" (UID: "2d3448ba-cd49-4b20-ae59-d1c0cb905d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:04.821982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.821952 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9" pod "2d3448ba-cd49-4b20-ae59-d1c0cb905d70" (UID: "2d3448ba-cd49-4b20-ae59-d1c0cb905d70"). InnerVolumeSpecName "kube-api-access-8wsh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:04.823176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.823150 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca" pod "2d3448ba-cd49-4b20-ae59-d1c0cb905d70" (UID: "2d3448ba-cd49-4b20-ae59-d1c0cb905d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:04.920393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.920299 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:04.920393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.920336 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsh9\" (UniqueName: \"kubernetes.io/projected/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-kube-api-access-8wsh9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:04.920393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:04.920353 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3448ba-cd49-4b20-ae59-d1c0cb905d70-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:05.364229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.363569 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:19:05.369223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.364616 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2d3448ba-cd49-4b20-ae59-d1c0cb905d70" containerName="revoke-keys" Oct 31 01:19:05.369223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.364645 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3448ba-cd49-4b20-ae59-d1c0cb905d70" containerName="revoke-keys" Oct 31 01:19:05.369223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.364791 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3448ba-cd49-4b20-ae59-d1c0cb905d70" containerName="revoke-keys" Oct 31 01:19:05.390244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.388512 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.418228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.405960 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d251f-dockercfg-zvrnb\"" Oct 31 01:19:05.418228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.406162 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d251f\"" Oct 31 01:19:05.436352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.436315 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:19:05.487100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.487054 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.487370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.487342 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f82j\" (UniqueName: \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.588729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.588695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f82j\" (UniqueName: \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.588934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.588826 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.589657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.589634 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.608388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.608354 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" Oct 31 01:19:05.608596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.608385 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t" event={"ID":"2d3448ba-cd49-4b20-ae59-d1c0cb905d70","Type":"ContainerDied","Data":"dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781"} Oct 31 01:19:05.608596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.608435 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dade446339fb3b3bb35bf670977626ea064a5bf11d75f995b0f2328a545b8781" Oct 31 01:19:05.612580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.612554 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f82j\" (UniqueName: \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:05.734273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:05.734244 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:06.051144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:06.051110 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:19:06.053547 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:19:06.053514 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c11b71_ec2d_45e1_b9bd_35a92945fd99.slice/crio-a6780fab36aa28fc6e322ac1d6750a1962ec1ce323e53ef949c679de87028b74 WatchSource:0}: Error finding container a6780fab36aa28fc6e322ac1d6750a1962ec1ce323e53ef949c679de87028b74: Status 404 returned error can't find the container with id a6780fab36aa28fc6e322ac1d6750a1962ec1ce323e53ef949c679de87028b74 Oct 31 01:19:06.618491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:06.618454 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" event={"ID":"34c11b71-ec2d-45e1-b9bd-35a92945fd99","Type":"ContainerStarted","Data":"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7"} Oct 31 01:19:06.618706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:06.618520 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" event={"ID":"34c11b71-ec2d-45e1-b9bd-35a92945fd99","Type":"ContainerStarted","Data":"a6780fab36aa28fc6e322ac1d6750a1962ec1ce323e53ef949c679de87028b74"} Oct 31 01:19:06.618706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:06.618553 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:06.664608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:06.664555 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" podStartSLOduration=1.664541136 podStartE2EDuration="1.664541136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:06.660143931 +0000 UTC m=+3615.109231571" watchObservedRunningTime="2026-10-31 01:19:06.664541136 +0000 UTC m=+3615.113628796" Oct 31 01:19:07.034755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:07.034661 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp"] Oct 31 01:19:07.041421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:07.041384 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f910c4-vf4vp"] Oct 31 01:19:07.683472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:07.683436 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7161847-a1d8-458b-8fbf-fb033b291ac3" path="/var/lib/kubelet/pods/d7161847-a1d8-458b-8fbf-fb033b291ac3/volumes" Oct 31 01:19:08.634921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:08.634885 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782"] Oct 31 01:19:08.646543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:08.646513 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-476a67-zg782"] Oct 31 01:19:09.684422 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:09.684387 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3" path="/var/lib/kubelet/pods/0a0c953a-5b72-4b9a-9c09-b4e9508b6dc3/volumes" Oct 31 01:19:11.483903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:11.483870 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:19:16.343402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.343351 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" podUID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" containerName="main" containerID="cri-o://f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d" gracePeriod=45 Oct 31 01:19:16.586817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.586781 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn"] Oct 31 01:19:16.591330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.591301 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.609307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.609276 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn"] Oct 31 01:19:16.654932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.654904 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:19:16.676153 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.676117 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:16.676320 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.676280 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:19:16.738072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738034 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.738317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738145 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.738317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.738317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738309 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\" (UniqueName: \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.738530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738452 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\" (UniqueName: \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.738530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.738489 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\") pod \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\" (UID: \"4877aa36-7654-4e4e-9f54-ae0dfd5b6393\") " Oct 31 01:19:16.739052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739020 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:16.739156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739049 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.739350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739325 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:16.739428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739383 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqmr\" (UniqueName: \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.739428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739420 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.739601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739580 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:16.739668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739619 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.739668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.739638 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.741043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.741014 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:16.742041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.742012 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "kube-api-access-rt9fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:16.742179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.742154 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs" pod "4877aa36-7654-4e4e-9f54-ae0dfd5b6393" (UID: "4877aa36-7654-4e4e-9f54-ae0dfd5b6393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:16.760085 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.760063 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" Oct 31 01:19:16.760244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.760083 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" event={"ID":"4877aa36-7654-4e4e-9f54-ae0dfd5b6393","Type":"ContainerDied","Data":"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d"} Oct 31 01:19:16.760244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.760139 5011 scope.go:122] "RemoveContainer" containerID="f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d" Oct 31 01:19:16.768477 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.768436 5011 generic.go:356] "Generic (PLEG): container finished" podID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" containerID="f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d" exitCode=0 Oct 31 01:19:16.768590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.768564 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2" event={"ID":"4877aa36-7654-4e4e-9f54-ae0dfd5b6393","Type":"ContainerDied","Data":"6dd0a09c3751107bb726f355f98e2327bdaf9d1c98b77f57da91e2fcd82f8fee"} Oct 31 01:19:16.779344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.779322 5011 scope.go:122] "RemoveContainer" containerID="f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d" Oct 31 01:19:16.779624 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:19:16.779601 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d\": container with ID starting with f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d not found: ID does not exist" containerID="f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d" Oct 31 01:19:16.779679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.779637 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d"} err="failed to get container status \"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d\": rpc error: code = NotFound desc = could not find container \"f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d\": container with ID starting with f244c3d9bafa3562313ba5d274f28048304dd479d6684b1d95daf96efd389e1d not found: ID does not exist" Oct 31 01:19:16.815494 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.815470 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:19:16.821870 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.821846 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3c90d8/sub-test-model-3c90d8-kserve-84c6474dd7-fgqs2"] Oct 31 01:19:16.840324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840300 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqmr\" (UniqueName: \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.840452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840332 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.840527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840476 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.840607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840584 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.840607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840602 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.840733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840615 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.840733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.840626 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9fj\" (UniqueName: \"kubernetes.io/projected/4877aa36-7654-4e4e-9f54-ae0dfd5b6393-kube-api-access-rt9fj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:16.841362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.841316 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.842902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.842878 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.863582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.863529 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqmr\" (UniqueName: \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:16.908359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:16.908331 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:17.223113 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:19:17.223074 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c66ffe_5ae4_4da8_9896_53d3064c8c6c.slice/crio-7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a WatchSource:0}: Error finding container 7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a: Status 404 returned error can't find the container with id 7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a Oct 31 01:19:17.224939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.224914 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn"] Oct 31 01:19:17.636585 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.636558 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:19:17.685593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.685561 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" path="/var/lib/kubelet/pods/4877aa36-7654-4e4e-9f54-ae0dfd5b6393/volumes" Oct 31 01:19:17.782851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.782814 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" event={"ID":"47c66ffe-5ae4-4da8-9896-53d3064c8c6c","Type":"ContainerStarted","Data":"74fedaa0034dac4f40c4266725cb6c80d63fae529f6a37f532b846cb986497b7"} Oct 31 01:19:17.783046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.782876 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" event={"ID":"47c66ffe-5ae4-4da8-9896-53d3064c8c6c","Type":"ContainerStarted","Data":"7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a"} Oct 31 01:19:17.849285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:17.849224 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" podStartSLOduration=1.849189132 podStartE2EDuration="1.849189132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:17.846534716 +0000 UTC m=+3626.295622356" watchObservedRunningTime="2026-10-31 01:19:17.849189132 +0000 UTC m=+3626.298276803" Oct 31 01:19:18.794602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:18.794565 5011 generic.go:356] "Generic (PLEG): container finished" podID="47c66ffe-5ae4-4da8-9896-53d3064c8c6c" containerID="74fedaa0034dac4f40c4266725cb6c80d63fae529f6a37f532b846cb986497b7" exitCode=0 Oct 31 01:19:18.795024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:18.794644 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" event={"ID":"47c66ffe-5ae4-4da8-9896-53d3064c8c6c","Type":"ContainerDied","Data":"74fedaa0034dac4f40c4266725cb6c80d63fae529f6a37f532b846cb986497b7"} Oct 31 01:19:19.971685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:19.971655 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:20.085356 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.085323 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\") pod \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " Oct 31 01:19:20.085598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.085477 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\") pod \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " Oct 31 01:19:20.085598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.085535 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\" (UniqueName: \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\") pod \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\" (UID: \"47c66ffe-5ae4-4da8-9896-53d3064c8c6c\") " Oct 31 01:19:20.085770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.085723 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca" pod "47c66ffe-5ae4-4da8-9896-53d3064c8c6c" (UID: "47c66ffe-5ae4-4da8-9896-53d3064c8c6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:20.086104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.086078 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:20.087757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.087727 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr" pod "47c66ffe-5ae4-4da8-9896-53d3064c8c6c" (UID: "47c66ffe-5ae4-4da8-9896-53d3064c8c6c"). InnerVolumeSpecName "kube-api-access-vtqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:20.087867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.087778 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca" pod "47c66ffe-5ae4-4da8-9896-53d3064c8c6c" (UID: "47c66ffe-5ae4-4da8-9896-53d3064c8c6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:20.188217 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.188173 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:20.188351 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.188241 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqmr\" (UniqueName: \"kubernetes.io/projected/47c66ffe-5ae4-4da8-9896-53d3064c8c6c-kube-api-access-vtqmr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:20.775638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.775603 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt"] Oct 31 01:19:20.776247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776195 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" containerName="main" Oct 31 01:19:20.776247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776223 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" containerName="main" Oct 31 01:19:20.776247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776244 5011 cpu_manager.go:414] "containerMap: removing container" podUID="47c66ffe-5ae4-4da8-9896-53d3064c8c6c" containerName="revoke-keys" Oct 31 01:19:20.776247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776249 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c66ffe-5ae4-4da8-9896-53d3064c8c6c" containerName="revoke-keys" Oct 31 01:19:20.776518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776353 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4877aa36-7654-4e4e-9f54-ae0dfd5b6393" containerName="main" Oct 31 01:19:20.776518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.776363 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c66ffe-5ae4-4da8-9896-53d3064c8c6c" containerName="revoke-keys" Oct 31 01:19:20.780019 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.779999 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:20.798992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.798971 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt"] Oct 31 01:19:20.816162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.816135 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" event={"ID":"47c66ffe-5ae4-4da8-9896-53d3064c8c6c","Type":"ContainerDied","Data":"7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a"} Oct 31 01:19:20.816405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.816177 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cee4638935a899be2d5d1771e235098dc3c5f3c51ab533c8e1a7d764a4d2c6a" Oct 31 01:19:20.816405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.816146 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn" Oct 31 01:19:20.899785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.899749 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:20.899934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.899897 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:20.900101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:20.900079 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxp5\" (UniqueName: \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.001777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.001736 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.002292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.001861 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.002292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.002032 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxp5\" (UniqueName: \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.002498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.002475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.004394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.004370 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.025247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.025184 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxp5\" (UniqueName: \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.094316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.094289 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:21.425534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.425504 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt"] Oct 31 01:19:21.826775 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:21.826737 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" event={"ID":"2e7c4bf7-5522-435c-9622-a2a572d1f36a","Type":"ContainerStarted","Data":"9ed40f6f174c27220d16b51f2ce8982fd7d1f6a3b7e4dcd80ca9862d25b2178b"} Oct 31 01:19:22.836486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:22.836455 5011 generic.go:356] "Generic (PLEG): container finished" podID="2e7c4bf7-5522-435c-9622-a2a572d1f36a" containerID="f8c736c6a53b0977d7e3e00af2acdea602872c41f342b15ada680ce02f5c764b" exitCode=0 Oct 31 01:19:22.836877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:22.836538 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" event={"ID":"2e7c4bf7-5522-435c-9622-a2a572d1f36a","Type":"ContainerDied","Data":"f8c736c6a53b0977d7e3e00af2acdea602872c41f342b15ada680ce02f5c764b"} Oct 31 01:19:24.012892 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.012864 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:24.048336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.048307 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\") pod \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " Oct 31 01:19:24.048507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.048484 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\" (UniqueName: \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\") pod \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " Oct 31 01:19:24.048641 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.048551 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\") pod \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\" (UID: \"2e7c4bf7-5522-435c-9622-a2a572d1f36a\") " Oct 31 01:19:24.048964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.048935 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca" pod "2e7c4bf7-5522-435c-9622-a2a572d1f36a" (UID: "2e7c4bf7-5522-435c-9622-a2a572d1f36a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:24.050551 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.050525 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca" pod "2e7c4bf7-5522-435c-9622-a2a572d1f36a" (UID: "2e7c4bf7-5522-435c-9622-a2a572d1f36a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:24.050643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.050541 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5" pod "2e7c4bf7-5522-435c-9622-a2a572d1f36a" (UID: "2e7c4bf7-5522-435c-9622-a2a572d1f36a"). InnerVolumeSpecName "kube-api-access-4nxp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:24.151025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.150948 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxp5\" (UniqueName: \"kubernetes.io/projected/2e7c4bf7-5522-435c-9622-a2a572d1f36a-kube-api-access-4nxp5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:24.151025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.150983 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:24.151025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.151000 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7c4bf7-5522-435c-9622-a2a572d1f36a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:24.856294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.856255 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" event={"ID":"2e7c4bf7-5522-435c-9622-a2a572d1f36a","Type":"ContainerDied","Data":"9ed40f6f174c27220d16b51f2ce8982fd7d1f6a3b7e4dcd80ca9862d25b2178b"} Oct 31 01:19:24.856504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.856315 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed40f6f174c27220d16b51f2ce8982fd7d1f6a3b7e4dcd80ca9862d25b2178b" Oct 31 01:19:24.856504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:24.856274 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt" Oct 31 01:19:31.949649 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:19:31.949610 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:19:38.705190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:38.699399 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 01:19:38.705190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:38.699636 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" podUID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" containerName="maas-api" containerID="cri-o://a967e5a191830536818391a602225637df24ad546fe20efa3fd7759fb29e122b" gracePeriod=30 Oct 31 01:19:39.305139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.304552 5011 generic.go:356] "Generic (PLEG): container finished" podID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" containerID="a967e5a191830536818391a602225637df24ad546fe20efa3fd7759fb29e122b" exitCode=0 Oct 31 01:19:39.305139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.304629 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" event={"ID":"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc","Type":"ContainerDied","Data":"a967e5a191830536818391a602225637df24ad546fe20efa3fd7759fb29e122b"} Oct 31 01:19:39.323675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.323645 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 01:19:39.422898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.422856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") pod \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " Oct 31 01:19:39.423121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.423091 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\" (UniqueName: \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\") pod \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " Oct 31 01:19:39.423331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.423140 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\") pod \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " Oct 31 01:19:39.423331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.423267 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\") pod \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\" (UID: \"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc\") " Oct 31 01:19:39.423890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.423863 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca" pod "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" (UID: "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:39.432624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.432578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls" pod "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" (UID: "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:39.433051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.433022 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p" pod "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" (UID: "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc"). InnerVolumeSpecName "kube-api-access-lv54p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:39.433945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.433918 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs" pod "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" (UID: "b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:39.461687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.461655 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t"] Oct 31 01:19:39.476582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.476553 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-498580-q7n6t"] Oct 31 01:19:39.525663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.525586 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv54p\" (UniqueName: \"kubernetes.io/projected/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-kube-api-access-lv54p\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:39.525663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.525615 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:39.525663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.525629 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:39.525663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.525641 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:39.684679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:39.684642 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3448ba-cd49-4b20-ae59-d1c0cb905d70" path="/var/lib/kubelet/pods/2d3448ba-cd49-4b20-ae59-d1c0cb905d70/volumes" Oct 31 01:19:40.313772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:40.313741 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" Oct 31 01:19:40.313772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:40.313758 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv" event={"ID":"b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc","Type":"ContainerDied","Data":"73ed46ca9f59f88185a249a3736b0c42406acebd18e102cf150f3c7e101a5e15"} Oct 31 01:19:40.314393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:40.313822 5011 scope.go:122] "RemoveContainer" containerID="a967e5a191830536818391a602225637df24ad546fe20efa3fd7759fb29e122b" Oct 31 01:19:40.367119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:40.367086 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 01:19:40.379455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:40.379424 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-498580-97c68dd-fmngv"] Oct 31 01:19:41.684271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:41.684201 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" path="/var/lib/kubelet/pods/b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc/volumes" Oct 31 01:19:46.653997 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:46.653958 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:19:46.654992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:46.654955 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" containerID="cri-o://88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed" gracePeriod=30 Oct 31 01:19:47.336441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.336408 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx"] Oct 31 01:19:47.337003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.336988 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2e7c4bf7-5522-435c-9622-a2a572d1f36a" containerName="revoke-keys" Oct 31 01:19:47.337092 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.337003 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7c4bf7-5522-435c-9622-a2a572d1f36a" containerName="revoke-keys" Oct 31 01:19:47.337092 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.337034 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" containerName="maas-api" Oct 31 01:19:47.337092 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.337039 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" containerName="maas-api" Oct 31 01:19:47.337302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.337123 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fb6503-bf9c-4835-bfe6-7f35b7ed85bc" containerName="maas-api" Oct 31 01:19:47.337302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.337138 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7c4bf7-5522-435c-9622-a2a572d1f36a" containerName="revoke-keys" Oct 31 01:19:47.342238 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.342218 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.360602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.360395 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:47.360602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.360531 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:19:47.366260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.366233 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx"] Oct 31 01:19:47.434456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.434426 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.434632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.434553 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.434730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.434703 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlf2\" (UniqueName: \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.536039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.535999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.536254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.536125 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.536321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.536268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlf2\" (UniqueName: \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.536950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.536918 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.538838 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.538811 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.560190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.560162 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlf2\" (UniqueName: \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.657190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.657113 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:47.966169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:47.966135 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx"] Oct 31 01:19:47.967314 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:19:47.967279 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd119e6f_731e_401a_ac21_7eeb064e79d5.slice/crio-68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7 WatchSource:0}: Error finding container 68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7: Status 404 returned error can't find the container with id 68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7 Oct 31 01:19:48.441341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:48.441273 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" event={"ID":"cd119e6f-731e-401a-ac21-7eeb064e79d5","Type":"ContainerStarted","Data":"68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7"} Oct 31 01:19:49.459026 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:49.458985 5011 generic.go:356] "Generic (PLEG): container finished" podID="cd119e6f-731e-401a-ac21-7eeb064e79d5" containerID="8ea42091a3a33ee42eaf7aaa387ebac8cb534ba1ae9c2ca77b91fb4eebfaba19" exitCode=0 Oct 31 01:19:49.459543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:49.459125 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" event={"ID":"cd119e6f-731e-401a-ac21-7eeb064e79d5","Type":"ContainerDied","Data":"8ea42091a3a33ee42eaf7aaa387ebac8cb534ba1ae9c2ca77b91fb4eebfaba19"} Oct 31 01:19:50.635889 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.635851 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:50.699311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.699271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\") pod \"cd119e6f-731e-401a-ac21-7eeb064e79d5\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " Oct 31 01:19:50.699511 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.699496 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\") pod \"cd119e6f-731e-401a-ac21-7eeb064e79d5\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " Oct 31 01:19:50.699586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.699565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\" (UniqueName: \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\") pod \"cd119e6f-731e-401a-ac21-7eeb064e79d5\" (UID: \"cd119e6f-731e-401a-ac21-7eeb064e79d5\") " Oct 31 01:19:50.699735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.699704 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca" pod "cd119e6f-731e-401a-ac21-7eeb064e79d5" (UID: "cd119e6f-731e-401a-ac21-7eeb064e79d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:50.700106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.700083 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:50.702521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.702483 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca" pod "cd119e6f-731e-401a-ac21-7eeb064e79d5" (UID: "cd119e6f-731e-401a-ac21-7eeb064e79d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:50.702800 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.702773 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2" pod "cd119e6f-731e-401a-ac21-7eeb064e79d5" (UID: "cd119e6f-731e-401a-ac21-7eeb064e79d5"). InnerVolumeSpecName "kube-api-access-mzlf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:50.801799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.801737 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd119e6f-731e-401a-ac21-7eeb064e79d5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:50.801799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:50.801759 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlf2\" (UniqueName: \"kubernetes.io/projected/cd119e6f-731e-401a-ac21-7eeb064e79d5-kube-api-access-mzlf2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:51.478964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:51.478935 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" Oct 31 01:19:51.479165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:51.478971 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx" event={"ID":"cd119e6f-731e-401a-ac21-7eeb064e79d5","Type":"ContainerDied","Data":"68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7"} Oct 31 01:19:51.479165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:51.479031 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68fb49de7ec2619444912cd2679fa09d41916497e9c9f860ee20367b0f92aec7" Oct 31 01:19:51.989910 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:51.989884 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:19:52.116576 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116539 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.116844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116607 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.116844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116764 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.116844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116832 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116867 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116888 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116918 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.116953 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.117045 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\") pod \"e74c9431-541e-4105-9950-20b2b24b50f8\" (UID: \"e74c9431-541e-4105-9950-20b2b24b50f8\") " Oct 31 01:19:52.117876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.117850 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:52.118134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.118070 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:52.118366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.118269 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:52.118578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.118553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:52.118816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.118796 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:52.120037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.120009 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:52.120159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.120140 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:52.120267 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.120147 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:19:52.121759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.121736 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk" pod "e74c9431-541e-4105-9950-20b2b24b50f8" (UID: "e74c9431-541e-4105-9950-20b2b24b50f8"). InnerVolumeSpecName "kube-api-access-4k9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:52.218940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218910 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.218940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218937 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218947 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9dk\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-kube-api-access-4k9dk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218957 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218966 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218976 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74c9431-541e-4105-9950-20b2b24b50f8-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218984 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74c9431-541e-4105-9950-20b2b24b50f8-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218991 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74c9431-541e-4105-9950-20b2b24b50f8-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.219106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.218999 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74c9431-541e-4105-9950-20b2b24b50f8-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:19:52.489259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.489159 5011 generic.go:356] "Generic (PLEG): container finished" podID="e74c9431-541e-4105-9950-20b2b24b50f8" containerID="88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed" exitCode=0 Oct 31 01:19:52.489259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.489245 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" Oct 31 01:19:52.489498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.489258 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" event={"ID":"e74c9431-541e-4105-9950-20b2b24b50f8","Type":"ContainerDied","Data":"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed"} Oct 31 01:19:52.489498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.489319 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5" event={"ID":"e74c9431-541e-4105-9950-20b2b24b50f8","Type":"ContainerDied","Data":"5058002db3221e32c4e63c1c146efa52d468671d18c778b1ef903c13b69b50b3"} Oct 31 01:19:52.489498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.489342 5011 scope.go:122] "RemoveContainer" containerID="88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed" Oct 31 01:19:52.501090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.501068 5011 scope.go:122] "RemoveContainer" containerID="88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed" Oct 31 01:19:52.501407 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:19:52.501386 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed\": container with ID starting with 88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed not found: ID does not exist" containerID="88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed" Oct 31 01:19:52.501486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.501415 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed"} err="failed to get container status \"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed\": rpc error: code = NotFound desc = could not find container \"88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed\": container with ID starting with 88a0abbd90f2ef6984dc1199b1706f33d800f83cff9d45dd0fa77a7a6fe0cbed not found: ID does not exist" Oct 31 01:19:52.543387 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.543357 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:19:52.555734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:52.555701 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-4pff5"] Oct 31 01:19:53.683918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:53.683880 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" path="/var/lib/kubelet/pods/e74c9431-541e-4105-9950-20b2b24b50f8/volumes" Oct 31 01:19:53.877165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:53.877131 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn"] Oct 31 01:19:53.900003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:53.890300 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b8fd5a-tdkgn"] Oct 31 01:19:54.983359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:54.983325 5011 scope.go:122] "RemoveContainer" containerID="756f01bdc41f7a755bca5ff114effb8843e59bbd05a9d3b2f31cf7dc04e66f08" Oct 31 01:19:54.995285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:54.995261 5011 scope.go:122] "RemoveContainer" containerID="055e0e0d606588a499982765d876ba1f8d904b43b09d36aacd7b5d5147cdcf49" Oct 31 01:19:55.684653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:19:55.684614 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c66ffe-5ae4-4da8-9896-53d3064c8c6c" path="/var/lib/kubelet/pods/47c66ffe-5ae4-4da8-9896-53d3064c8c6c/volumes" Oct 31 01:20:01.073271 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.073230 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:20:01.073932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.073898 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" containerID="cri-o://207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48" gracePeriod=30 Oct 31 01:20:01.859124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859087 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h"] Oct 31 01:20:01.859745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859726 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" Oct 31 01:20:01.859745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859744 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" Oct 31 01:20:01.859902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859755 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cd119e6f-731e-401a-ac21-7eeb064e79d5" containerName="revoke-keys" Oct 31 01:20:01.859902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859760 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd119e6f-731e-401a-ac21-7eeb064e79d5" containerName="revoke-keys" Oct 31 01:20:01.859902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859847 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74c9431-541e-4105-9950-20b2b24b50f8" containerName="istio-proxy" Oct 31 01:20:01.859902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.859855 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd119e6f-731e-401a-ac21-7eeb064e79d5" containerName="revoke-keys" Oct 31 01:20:01.863693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.863666 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.881979 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.881952 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h"] Oct 31 01:20:01.895145 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.895118 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.895457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.895438 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2vk\" (UniqueName: \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.895597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.895476 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.906183 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.906157 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:01.906328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.906306 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:01.954347 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:01.954306 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:20:01.996962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.996932 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2vk\" (UniqueName: \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.997165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.996974 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.997165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.997007 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.997723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.997703 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:01.999434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:01.999412 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:02.020825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:02.020795 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2vk\" (UniqueName: \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\") pod \"maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:02.182075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:02.181998 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:02.511913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:02.511878 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h"] Oct 31 01:20:02.514074 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:02.514045 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadde956_9fe6_48ff_8fe4_5ec94c580cc1.slice/crio-5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4 WatchSource:0}: Error finding container 5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4: Status 404 returned error can't find the container with id 5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4 Oct 31 01:20:02.652532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:02.652489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" event={"ID":"eadde956-9fe6-48ff-8fe4-5ec94c580cc1","Type":"ContainerStarted","Data":"5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4"} Oct 31 01:20:03.423492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:03.423455 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt"] Oct 31 01:20:03.435337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:03.435305 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa97ea-5z7qt"] Oct 31 01:20:03.664186 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:03.664149 5011 generic.go:356] "Generic (PLEG): container finished" podID="eadde956-9fe6-48ff-8fe4-5ec94c580cc1" containerID="1e45626161a57ad72af3f09dc74b68afc17b5c9790a4bcec075f8f91b320657d" exitCode=0 Oct 31 01:20:03.664353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:03.664229 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" event={"ID":"eadde956-9fe6-48ff-8fe4-5ec94c580cc1","Type":"ContainerDied","Data":"1e45626161a57ad72af3f09dc74b68afc17b5c9790a4bcec075f8f91b320657d"} Oct 31 01:20:03.683075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:03.683013 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7c4bf7-5522-435c-9622-a2a572d1f36a" path="/var/lib/kubelet/pods/2e7c4bf7-5522-435c-9622-a2a572d1f36a/volumes" Oct 31 01:20:04.835022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.834996 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:04.861732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.861704 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\") pod \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " Oct 31 01:20:04.861885 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.861869 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\") pod \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " Oct 31 01:20:04.861965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.861938 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\" (UniqueName: \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\") pod \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\" (UID: \"eadde956-9fe6-48ff-8fe4-5ec94c580cc1\") " Oct 31 01:20:04.862107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.862079 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca" pod "eadde956-9fe6-48ff-8fe4-5ec94c580cc1" (UID: "eadde956-9fe6-48ff-8fe4-5ec94c580cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:04.862507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.862475 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:04.864505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.864473 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca" pod "eadde956-9fe6-48ff-8fe4-5ec94c580cc1" (UID: "eadde956-9fe6-48ff-8fe4-5ec94c580cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:04.864505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.864487 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk" pod "eadde956-9fe6-48ff-8fe4-5ec94c580cc1" (UID: "eadde956-9fe6-48ff-8fe4-5ec94c580cc1"). InnerVolumeSpecName "kube-api-access-rl2vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:04.963291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.963189 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:04.963291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:04.963239 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2vk\" (UniqueName: \"kubernetes.io/projected/eadde956-9fe6-48ff-8fe4-5ec94c580cc1-kube-api-access-rl2vk\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:05.682072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:05.682040 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" Oct 31 01:20:05.683610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:05.683579 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h" event={"ID":"eadde956-9fe6-48ff-8fe4-5ec94c580cc1","Type":"ContainerDied","Data":"5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4"} Oct 31 01:20:05.683738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:05.683634 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e12d8e8588506f94ef32972ab469928f95d433376854c063309809ef769d1a4" Oct 31 01:20:06.387693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.387668 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:20:06.482183 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482152 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482235 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482396 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482442 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482492 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482527 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482578 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:06.482772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482607 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482651 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\") pod \"0b878d08-8de0-4f73-9b8e-f0bffd236715\" (UID: \"0b878d08-8de0-4f73-9b8e-f0bffd236715\") " Oct 31 01:20:06.482772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482676 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:06.482772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482745 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:06.482976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.482956 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:06.483056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:06.483340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483320 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.483409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483346 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.483409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483360 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.483409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483375 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.483409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.483389 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878d08-8de0-4f73-9b8e-f0bffd236715-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.484851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.484829 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:06.484851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.484830 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:06.485165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.485149 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "kube-api-access-vl8qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:06.485402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.485373 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy" pod "0b878d08-8de0-4f73-9b8e-f0bffd236715" (UID: "0b878d08-8de0-4f73-9b8e-f0bffd236715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:06.572657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.572618 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 01:20:06.584253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.584185 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 01:20:06.585155 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.585125 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.585348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.585162 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8qh\" (UniqueName: \"kubernetes.io/projected/0b878d08-8de0-4f73-9b8e-f0bffd236715-kube-api-access-vl8qh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.585348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.585179 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.585348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.585193 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878d08-8de0-4f73-9b8e-f0bffd236715-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:06.696679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.696579 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerID="207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48" exitCode=0 Oct 31 01:20:06.696679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.696626 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" Oct 31 01:20:06.696679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.696650 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerDied","Data":"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48"} Oct 31 01:20:06.696973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.696705 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w" event={"ID":"0b878d08-8de0-4f73-9b8e-f0bffd236715","Type":"ContainerDied","Data":"7742a792a5b2a41993660fa746cfb513ddd4eaaa9022008ce8215cd1d0fff4a0"} Oct 31 01:20:06.696973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.696735 5011 scope.go:122] "RemoveContainer" containerID="207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48" Oct 31 01:20:06.710593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.710556 5011 scope.go:122] "RemoveContainer" containerID="504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074" Oct 31 01:20:06.721670 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.721650 5011 scope.go:122] "RemoveContainer" containerID="207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48" Oct 31 01:20:06.721931 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:06.721911 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48\": container with ID starting with 207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48 not found: ID does not exist" containerID="207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48" Oct 31 01:20:06.722012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.721938 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48"} err="failed to get container status \"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48\": rpc error: code = NotFound desc = could not find container \"207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48\": container with ID starting with 207b110142c99138df80a348ae12e445ac6dd49482856fce5ee4b3ca783c8c48 not found: ID does not exist" Oct 31 01:20:06.722012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.721962 5011 scope.go:122] "RemoveContainer" containerID="504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074" Oct 31 01:20:06.722197 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:06.722180 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074\": container with ID starting with 504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074 not found: ID does not exist" containerID="504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074" Oct 31 01:20:06.722293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.722201 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074"} err="failed to get container status \"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074\": rpc error: code = NotFound desc = could not find container \"504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074\": container with ID starting with 504a39f9561fa6a34996fa10c29576dc97046c3b2f7593e92f0a8bcb759d0074 not found: ID does not exist" Oct 31 01:20:06.753594 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.753569 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:20:06.763732 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:06.763698 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-b8fd5a-openshift-default-f9bcbdd7-89h5w"] Oct 31 01:20:07.683919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:07.683882 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" path="/var/lib/kubelet/pods/0b878d08-8de0-4f73-9b8e-f0bffd236715/volumes" Oct 31 01:20:10.673458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:10.672443 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:20:10.673458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:10.672728 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" containerID="cri-o://a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa" gracePeriod=30 Oct 31 01:20:11.266413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.266375 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj"] Oct 31 01:20:11.267297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267277 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eadde956-9fe6-48ff-8fe4-5ec94c580cc1" containerName="revoke-keys" Oct 31 01:20:11.267297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267297 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eadde956-9fe6-48ff-8fe4-5ec94c580cc1" containerName="revoke-keys" Oct 31 01:20:11.267418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267320 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.267418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267326 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.267418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267334 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.267418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267341 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.267572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267452 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eadde956-9fe6-48ff-8fe4-5ec94c580cc1" containerName="revoke-keys" Oct 31 01:20:11.267572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267464 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.267572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.267470 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b878d08-8de0-4f73-9b8e-f0bffd236715" containerName="istio-proxy" Oct 31 01:20:11.272663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.272646 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.286855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.286827 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj"] Oct 31 01:20:11.366182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.366149 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.366359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.366199 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpb4\" (UniqueName: \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.366406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.366374 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.377561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.377534 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:11.377703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.377685 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:11.467324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.467288 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.467507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.467352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpb4\" (UniqueName: \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.467507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.467484 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.467987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.467961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.469866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.469846 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.491124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.491094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpb4\" (UniqueName: \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\") pod \"maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.587809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.587775 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:11.927156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.927012 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj"] Oct 31 01:20:11.932544 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:11.932512 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf795ced0_c7fa_4574_a080_2f681257390f.slice/crio-e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea WatchSource:0}: Error finding container e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea: Status 404 returned error can't find the container with id e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea Oct 31 01:20:11.945107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:11.945074 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:20:12.755421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:12.755385 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" event={"ID":"f795ced0-c7fa-4574-a080-2f681257390f","Type":"ContainerStarted","Data":"0d8c5314ba35e3e08e09ed5b0f02334a6b74d7adc50d82a5b3533fa75da2c77a"} Oct 31 01:20:12.755615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:12.755448 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" event={"ID":"f795ced0-c7fa-4574-a080-2f681257390f","Type":"ContainerStarted","Data":"e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea"} Oct 31 01:20:12.800015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:12.799963 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" podStartSLOduration=1.799949218 podStartE2EDuration="1.799949218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:12.79613096 +0000 UTC m=+3681.245218599" watchObservedRunningTime="2026-10-31 01:20:12.799949218 +0000 UTC m=+3681.249036856" Oct 31 01:20:13.764498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:13.764463 5011 generic.go:356] "Generic (PLEG): container finished" podID="f795ced0-c7fa-4574-a080-2f681257390f" containerID="0d8c5314ba35e3e08e09ed5b0f02334a6b74d7adc50d82a5b3533fa75da2c77a" exitCode=0 Oct 31 01:20:13.764981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:13.764506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" event={"ID":"f795ced0-c7fa-4574-a080-2f681257390f","Type":"ContainerDied","Data":"0d8c5314ba35e3e08e09ed5b0f02334a6b74d7adc50d82a5b3533fa75da2c77a"} Oct 31 01:20:14.939057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:14.939029 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:15.036359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.036281 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\" (UniqueName: \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\") pod \"f795ced0-c7fa-4574-a080-2f681257390f\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " Oct 31 01:20:15.036527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.036366 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\") pod \"f795ced0-c7fa-4574-a080-2f681257390f\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " Oct 31 01:20:15.036527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.036407 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\") pod \"f795ced0-c7fa-4574-a080-2f681257390f\" (UID: \"f795ced0-c7fa-4574-a080-2f681257390f\") " Oct 31 01:20:15.036791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.036755 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca" pod "f795ced0-c7fa-4574-a080-2f681257390f" (UID: "f795ced0-c7fa-4574-a080-2f681257390f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:15.038600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.038574 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4" pod "f795ced0-c7fa-4574-a080-2f681257390f" (UID: "f795ced0-c7fa-4574-a080-2f681257390f"). InnerVolumeSpecName "kube-api-access-5jpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:15.038700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.038640 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca" pod "f795ced0-c7fa-4574-a080-2f681257390f" (UID: "f795ced0-c7fa-4574-a080-2f681257390f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:15.137850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.137817 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpb4\" (UniqueName: \"kubernetes.io/projected/f795ced0-c7fa-4574-a080-2f681257390f-kube-api-access-5jpb4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:15.137850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.137844 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:15.137850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.137855 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f795ced0-c7fa-4574-a080-2f681257390f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:15.780710 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.780681 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" Oct 31 01:20:15.780969 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.780680 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj" event={"ID":"f795ced0-c7fa-4574-a080-2f681257390f","Type":"ContainerDied","Data":"e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea"} Oct 31 01:20:15.781799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.781023 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7188934b7afd7ab1212b883e76f60dc37a39387409ceb7165b0764c9f910aea" Oct 31 01:20:15.943053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:15.942996 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:20:16.048839 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.048805 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.048871 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.048911 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.048942 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049045 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049072 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:16.049255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049079 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049148 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049310 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\") pod \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\" (UID: \"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3\") " Oct 31 01:20:16.049544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049356 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:16.049544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049378 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:16.049738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:16.049914 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.049869 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:16.050077 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.050055 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.050367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.050082 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.050367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.050097 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.050367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.050112 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.050367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.050125 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.051438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.051414 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:16.051633 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.051574 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:16.051633 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.051580 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:16.051898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.051871 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8" pod "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" (UID: "e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3"). InnerVolumeSpecName "kube-api-access-f99w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:16.151926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.151891 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.152121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.151959 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.152121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.151975 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.152121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.151989 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99w8\" (UniqueName: \"kubernetes.io/projected/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3-kube-api-access-f99w8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:16.795717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.795676 5011 generic.go:356] "Generic (PLEG): container finished" podID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerID="a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa" exitCode=0 Oct 31 01:20:16.795978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.795725 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" Oct 31 01:20:16.795978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.795740 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerDied","Data":"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa"} Oct 31 01:20:16.795978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.795798 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4" event={"ID":"e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3","Type":"ContainerDied","Data":"ec14f1a1ce3159321f22974226a1a4dc9752bb456fbab647ad5cd47dfc63ba2f"} Oct 31 01:20:16.795978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.795829 5011 scope.go:122] "RemoveContainer" containerID="a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa" Oct 31 01:20:16.811071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.811045 5011 scope.go:122] "RemoveContainer" containerID="5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1" Oct 31 01:20:16.823274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.823247 5011 scope.go:122] "RemoveContainer" containerID="a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa" Oct 31 01:20:16.823557 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:16.823535 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa\": container with ID starting with a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa not found: ID does not exist" containerID="a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa" Oct 31 01:20:16.823644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.823566 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa"} err="failed to get container status \"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa\": rpc error: code = NotFound desc = could not find container \"a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa\": container with ID starting with a0fdeb3da756c531b767fb4bedb0c477ab05e33bfc762e941a386fa73034d2aa not found: ID does not exist" Oct 31 01:20:16.823644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.823584 5011 scope.go:122] "RemoveContainer" containerID="5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1" Oct 31 01:20:16.823836 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:16.823811 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1\": container with ID starting with 5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1 not found: ID does not exist" containerID="5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1" Oct 31 01:20:16.823926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.823842 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1"} err="failed to get container status \"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1\": rpc error: code = NotFound desc = could not find container \"5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1\": container with ID starting with 5ff6996395c3fb5e29c275d4c182fb6a4ab6676af22ca40c82c676476191d2c1 not found: ID does not exist" Oct 31 01:20:16.851076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.851047 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:20:16.863386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:16.863361 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa97ea-openshift-default-6b6c5d4f7-kpfn4"] Oct 31 01:20:17.685253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:17.685193 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" path="/var/lib/kubelet/pods/e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3/volumes" Oct 31 01:20:18.234111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:18.234074 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx"] Oct 31 01:20:18.245779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:18.245746 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3c90d8-fpwbx"] Oct 31 01:20:19.683401 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:19.683368 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd119e6f-731e-401a-ac21-7eeb064e79d5" path="/var/lib/kubelet/pods/cd119e6f-731e-401a-ac21-7eeb064e79d5/volumes" Oct 31 01:20:21.098018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.097978 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt"] Oct 31 01:20:21.098843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098825 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f795ced0-c7fa-4574-a080-2f681257390f" containerName="revoke-keys" Oct 31 01:20:21.098903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098845 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f795ced0-c7fa-4574-a080-2f681257390f" containerName="revoke-keys" Oct 31 01:20:21.098903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098875 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.098903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098884 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.099014 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098927 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.099014 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.098935 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.099087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.099052 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.099087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.099072 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a09d8a-7353-4172-a8a4-c7f76c5e6dc3" containerName="istio-proxy" Oct 31 01:20:21.099087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.099085 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f795ced0-c7fa-4574-a080-2f681257390f" containerName="revoke-keys" Oct 31 01:20:21.105467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.105442 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.122393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.122368 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt"] Oct 31 01:20:21.127170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.127023 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:21.127170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.127156 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:21.229355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.229318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5ph\" (UniqueName: \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.229536 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.229513 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.229666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.229647 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.330919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.330884 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.331120 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.331013 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.331120 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.331106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5ph\" (UniqueName: \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.331731 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.331704 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.333396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.333372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.354767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.354707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5ph\" (UniqueName: \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\") pod \"maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.428379 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.428354 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:21.597640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.597597 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" containerID="cri-o://599fbcc77c9679bfc9a9fa0d0850ab0413f6f455bd8209e5ab097f5d66cbcf4f" gracePeriod=45 Oct 31 01:20:21.610671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.610606 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" containerID="cri-o://d22c4a3c33e38b922998418ffa113d04f1322a0921fc4d9a528245b31693bf4d" gracePeriod=45 Oct 31 01:20:21.736580 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:21.736282 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf371aa21_b210_4b6e_8f94_e19620f301c3.slice/crio-0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571 WatchSource:0}: Error finding container 0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571: Status 404 returned error can't find the container with id 0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571 Oct 31 01:20:21.737243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.737115 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt"] Oct 31 01:20:21.874577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.874485 5011 generic.go:356] "Generic (PLEG): container finished" podID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerID="d22c4a3c33e38b922998418ffa113d04f1322a0921fc4d9a528245b31693bf4d" exitCode=0 Oct 31 01:20:21.874735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.874618 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" event={"ID":"91ef2907-6a05-4459-a039-ef9f1688b8e9","Type":"ContainerDied","Data":"d22c4a3c33e38b922998418ffa113d04f1322a0921fc4d9a528245b31693bf4d"} Oct 31 01:20:21.876717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.876686 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" event={"ID":"f371aa21-b210-4b6e-8f94-e19620f301c3","Type":"ContainerStarted","Data":"0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571"} Oct 31 01:20:21.878808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.878781 5011 generic.go:356] "Generic (PLEG): container finished" podID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerID="599fbcc77c9679bfc9a9fa0d0850ab0413f6f455bd8209e5ab097f5d66cbcf4f" exitCode=0 Oct 31 01:20:21.878983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.878871 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" event={"ID":"1c04352b-c078-45e6-bc97-110fb00e5d37","Type":"ContainerDied","Data":"599fbcc77c9679bfc9a9fa0d0850ab0413f6f455bd8209e5ab097f5d66cbcf4f"} Oct 31 01:20:21.939780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.939754 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 01:20:21.942386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:21.942366 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 01:20:22.039255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039189 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039338 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039384 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039416 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.039648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039455 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.039648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039463 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.039648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\" (UniqueName: \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.039899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039678 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.039899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039726 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039841 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.039899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039872 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\" (UniqueName: \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\") pod \"1c04352b-c078-45e6-bc97-110fb00e5d37\" (UID: \"1c04352b-c078-45e6-bc97-110fb00e5d37\") " Oct 31 01:20:22.039899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039850 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.040187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039900 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.040187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.039977 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\") pod \"91ef2907-6a05-4459-a039-ef9f1688b8e9\" (UID: \"91ef2907-6a05-4459-a039-ef9f1688b8e9\") " Oct 31 01:20:22.040187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.040397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040322 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.040646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040628 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.040741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040688 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.040741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040706 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.040741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040720 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.040741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040732 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.040977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.040935 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.043355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.043323 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.043490 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.043352 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "kube-api-access-jkq46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:22.043874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.043847 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "kube-api-access-cw5rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:22.045965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.045939 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs" pod "1c04352b-c078-45e6-bc97-110fb00e5d37" (UID: "1c04352b-c078-45e6-bc97-110fb00e5d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:22.046506 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.046479 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:22.047426 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.047405 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm" pod "91ef2907-6a05-4459-a039-ef9f1688b8e9" (UID: "91ef2907-6a05-4459-a039-ef9f1688b8e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.141973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.141944 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkq46\" (UniqueName: \"kubernetes.io/projected/91ef2907-6a05-4459-a039-ef9f1688b8e9-kube-api-access-jkq46\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.141979 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.141990 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5rd\" (UniqueName: \"kubernetes.io/projected/1c04352b-c078-45e6-bc97-110fb00e5d37-kube-api-access-cw5rd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.142001 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2907-6a05-4459-a039-ef9f1688b8e9-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.142009 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c04352b-c078-45e6-bc97-110fb00e5d37-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.142017 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c04352b-c078-45e6-bc97-110fb00e5d37-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.142410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.142024 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2907-6a05-4459-a039-ef9f1688b8e9-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.887362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.887327 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" Oct 31 01:20:22.887528 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.887356 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh" event={"ID":"91ef2907-6a05-4459-a039-ef9f1688b8e9","Type":"ContainerDied","Data":"676a037d92326ee54a018d7e109e0aa36097eb6036be51be36cca9b4d7dfd84b"} Oct 31 01:20:22.887528 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.887421 5011 scope.go:122] "RemoveContainer" containerID="d22c4a3c33e38b922998418ffa113d04f1322a0921fc4d9a528245b31693bf4d" Oct 31 01:20:22.889268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.889241 5011 generic.go:356] "Generic (PLEG): container finished" podID="f371aa21-b210-4b6e-8f94-e19620f301c3" containerID="c9cd41ded8f9db3d0d2a33a33d2ad803f750626dae6e8b350c2026d1b9676bfb" exitCode=0 Oct 31 01:20:22.889397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.889319 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" event={"ID":"f371aa21-b210-4b6e-8f94-e19620f301c3","Type":"ContainerDied","Data":"c9cd41ded8f9db3d0d2a33a33d2ad803f750626dae6e8b350c2026d1b9676bfb"} Oct 31 01:20:22.890919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.890902 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" Oct 31 01:20:22.891006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.890949 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf" event={"ID":"1c04352b-c078-45e6-bc97-110fb00e5d37","Type":"ContainerDied","Data":"4a31ad9c34e581591f066bb9440334b0636a9b3c899e5bf70d270149588876e7"} Oct 31 01:20:22.902568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.902538 5011 scope.go:122] "RemoveContainer" containerID="599fbcc77c9679bfc9a9fa0d0850ab0413f6f455bd8209e5ab097f5d66cbcf4f" Oct 31 01:20:22.943794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.943761 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 01:20:22.953490 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:22.953463 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/simulator-w1-b024b5-kserve-cbdd57ddc-wq9fh"] Oct 31 01:20:23.022464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:23.022391 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 01:20:23.031408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:23.031383 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/premium-w1-b024b5-kserve-5bc74d857d-8gxhf"] Oct 31 01:20:23.684355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:23.684318 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" path="/var/lib/kubelet/pods/1c04352b-c078-45e6-bc97-110fb00e5d37/volumes" Oct 31 01:20:23.684815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:23.684799 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" path="/var/lib/kubelet/pods/91ef2907-6a05-4459-a039-ef9f1688b8e9/volumes" Oct 31 01:20:24.065444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.065413 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:24.167591 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.167558 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\" (UniqueName: \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\") pod \"f371aa21-b210-4b6e-8f94-e19620f301c3\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " Oct 31 01:20:24.167749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.167650 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\") pod \"f371aa21-b210-4b6e-8f94-e19620f301c3\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " Oct 31 01:20:24.167834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.167755 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\") pod \"f371aa21-b210-4b6e-8f94-e19620f301c3\" (UID: \"f371aa21-b210-4b6e-8f94-e19620f301c3\") " Oct 31 01:20:24.168065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.168027 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca" pod "f371aa21-b210-4b6e-8f94-e19620f301c3" (UID: "f371aa21-b210-4b6e-8f94-e19620f301c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:24.169860 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.169828 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph" pod "f371aa21-b210-4b6e-8f94-e19620f301c3" (UID: "f371aa21-b210-4b6e-8f94-e19620f301c3"). InnerVolumeSpecName "kube-api-access-kv5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:24.169972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.169858 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca" pod "f371aa21-b210-4b6e-8f94-e19620f301c3" (UID: "f371aa21-b210-4b6e-8f94-e19620f301c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:24.269796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.269713 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:24.269796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.269749 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv5ph\" (UniqueName: \"kubernetes.io/projected/f371aa21-b210-4b6e-8f94-e19620f301c3-kube-api-access-kv5ph\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:24.269796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.269762 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f371aa21-b210-4b6e-8f94-e19620f301c3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:24.916175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.916145 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" Oct 31 01:20:24.916621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.916191 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt" event={"ID":"f371aa21-b210-4b6e-8f94-e19620f301c3","Type":"ContainerDied","Data":"0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571"} Oct 31 01:20:24.916621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:24.916264 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf50d7aa2cd73202fada2820feba966a651848e0c48507874de5e088d7b8571" Oct 31 01:20:25.763734 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:25.763685 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 01:20:25.764341 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:25.764310 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" containerID="cri-o://b60361993b37160bd36c64ff1491a885a0f1d4f4bc2004b0e6c2a953aecced0f" gracePeriod=30 Oct 31 01:20:26.511413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.511374 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst"] Oct 31 01:20:26.512152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512127 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f371aa21-b210-4b6e-8f94-e19620f301c3" containerName="revoke-keys" Oct 31 01:20:26.512337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512152 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f371aa21-b210-4b6e-8f94-e19620f301c3" containerName="revoke-keys" Oct 31 01:20:26.512337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512175 5011 cpu_manager.go:414] "containerMap: removing container" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" Oct 31 01:20:26.512337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512184 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" Oct 31 01:20:26.512337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512259 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" Oct 31 01:20:26.512337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512271 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" Oct 31 01:20:26.512607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512423 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c04352b-c078-45e6-bc97-110fb00e5d37" containerName="main" Oct 31 01:20:26.512607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512445 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f371aa21-b210-4b6e-8f94-e19620f301c3" containerName="revoke-keys" Oct 31 01:20:26.512607 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.512455 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef2907-6a05-4459-a039-ef9f1688b8e9" containerName="main" Oct 31 01:20:26.515975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.515954 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.534755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.534729 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst"] Oct 31 01:20:26.568768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.568740 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:26.569270 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.569199 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:26.596086 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.596059 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.596240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.596096 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjkh\" (UniqueName: \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.596412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.596390 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.697367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.697329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.697537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.697410 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.697537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.697437 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjkh\" (UniqueName: \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.698177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.698154 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.699923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.699896 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.720993 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.720961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjkh\" (UniqueName: \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\") pod \"maas-api-revoke-keys-e2e-worker-w0-671959-xfcst\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:26.834387 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:26.834358 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:27.144678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:27.144641 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst"] Oct 31 01:20:27.145052 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:27.145025 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f2ebc0_b4ff_4412_9bff_5a290e172884.slice/crio-794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392 WatchSource:0}: Error finding container 794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392: Status 404 returned error can't find the container with id 794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392 Oct 31 01:20:27.978484 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:27.978455 5011 generic.go:356] "Generic (PLEG): container finished" podID="24f2ebc0-b4ff-4412-9bff-5a290e172884" containerID="40826f583c0799ddfb59eae5292a29cf770441ecc08b20c4848f38a2795c9c67" exitCode=0 Oct 31 01:20:27.978802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:27.978542 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" event={"ID":"24f2ebc0-b4ff-4412-9bff-5a290e172884","Type":"ContainerDied","Data":"40826f583c0799ddfb59eae5292a29cf770441ecc08b20c4848f38a2795c9c67"} Oct 31 01:20:27.978802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:27.978601 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" event={"ID":"24f2ebc0-b4ff-4412-9bff-5a290e172884","Type":"ContainerStarted","Data":"794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392"} Oct 31 01:20:29.147827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.147794 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:29.229977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.229940 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\") pod \"24f2ebc0-b4ff-4412-9bff-5a290e172884\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " Oct 31 01:20:29.230172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.230008 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\") pod \"24f2ebc0-b4ff-4412-9bff-5a290e172884\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " Oct 31 01:20:29.230267 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.230171 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\" (UniqueName: \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\") pod \"24f2ebc0-b4ff-4412-9bff-5a290e172884\" (UID: \"24f2ebc0-b4ff-4412-9bff-5a290e172884\") " Oct 31 01:20:29.231198 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.231168 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca" pod "24f2ebc0-b4ff-4412-9bff-5a290e172884" (UID: "24f2ebc0-b4ff-4412-9bff-5a290e172884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:29.233248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.233177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca" pod "24f2ebc0-b4ff-4412-9bff-5a290e172884" (UID: "24f2ebc0-b4ff-4412-9bff-5a290e172884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:29.233356 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.233246 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh" pod "24f2ebc0-b4ff-4412-9bff-5a290e172884" (UID: "24f2ebc0-b4ff-4412-9bff-5a290e172884"). InnerVolumeSpecName "kube-api-access-7jjkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:29.332347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.332315 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:29.332347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.332344 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f2ebc0-b4ff-4412-9bff-5a290e172884-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:29.332347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.332355 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjkh\" (UniqueName: \"kubernetes.io/projected/24f2ebc0-b4ff-4412-9bff-5a290e172884-kube-api-access-7jjkh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:29.995160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.995080 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" Oct 31 01:20:29.995160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.995117 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst" event={"ID":"24f2ebc0-b4ff-4412-9bff-5a290e172884","Type":"ContainerDied","Data":"794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392"} Oct 31 01:20:29.995367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:29.995166 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794e5a4cf32ea32df57d1da47ec5a5beed4c7d0a0661a157ccc1d27a576a4392" Oct 31 01:20:31.008762 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.008723 5011 generic.go:356] "Generic (PLEG): container finished" podID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerID="b60361993b37160bd36c64ff1491a885a0f1d4f4bc2004b0e6c2a953aecced0f" exitCode=0 Oct 31 01:20:31.009131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.008763 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerDied","Data":"b60361993b37160bd36c64ff1491a885a0f1d4f4bc2004b0e6c2a953aecced0f"} Oct 31 01:20:31.009131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.008848 5011 scope.go:122] "RemoveContainer" containerID="43072c510fb38961ff94e7589fb2210cb2a217a74074c7b95fda947da511a5e3" Oct 31 01:20:31.039469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.039447 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 01:20:31.155640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.155603 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.155999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.155972 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156030 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156069 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156112 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156156 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156524 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156369 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\") pod \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\" (UID: \"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0\") " Oct 31 01:20:31.156932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.156907 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.157332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.157306 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:31.157434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.157310 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.158117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.158085 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.158435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.158416 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.160916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.160553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:31.167685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.167646 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.167685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.167667 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:31.170654 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.170625 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb" pod "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" (UID: "06d640c7-caa1-4ec8-ba7a-d66bf55a69b0"). InnerVolumeSpecName "kube-api-access-9xhkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:31.258650 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258610 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258650 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258644 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258650 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258653 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258665 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258674 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258683 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258690 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258699 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.258952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:31.258712 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhkb\" (UniqueName: \"kubernetes.io/projected/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0-kube-api-access-9xhkb\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.949758 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:31.949665 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:20:32.018029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.017991 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" event={"ID":"06d640c7-caa1-4ec8-ba7a-d66bf55a69b0","Type":"ContainerDied","Data":"1bc5b1cc35fc440e6c717c280691c8c08d7bc59499a2a58a3d36c229d21a56e2"} Oct 31 01:20:32.018029 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.018019 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r" Oct 31 01:20:32.018568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.018063 5011 scope.go:122] "RemoveContainer" containerID="b60361993b37160bd36c64ff1491a885a0f1d4f4bc2004b0e6c2a953aecced0f" Oct 31 01:20:32.036599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.036570 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x"] Oct 31 01:20:32.049233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.049194 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4162c32-z4s8x"] Oct 31 01:20:32.118737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.118701 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 01:20:32.126095 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:32.126066 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-8rn7r"] Oct 31 01:20:33.683318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:33.683282 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" path="/var/lib/kubelet/pods/06d640c7-caa1-4ec8-ba7a-d66bf55a69b0/volumes" Oct 31 01:20:33.683802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:33.683787 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc6754a-0245-470d-a3fd-d61d52de6ab4" path="/var/lib/kubelet/pods/4dc6754a-0245-470d-a3fd-d61d52de6ab4/volumes" Oct 31 01:20:42.034901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:42.034862 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt"] Oct 31 01:20:42.044729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:42.044703 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d28f0c-tw8rt"] Oct 31 01:20:43.684317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:43.684282 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f05b8af-531c-4ee1-94e2-11cba175a463" path="/var/lib/kubelet/pods/0f05b8af-531c-4ee1-94e2-11cba175a463/volumes" Oct 31 01:20:45.295388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.295347 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t"] Oct 31 01:20:45.296154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296128 5011 cpu_manager.go:414] "containerMap: removing container" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.296154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296154 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.296357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296195 5011 cpu_manager.go:414] "containerMap: removing container" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.296357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296220 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.296357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296271 5011 cpu_manager.go:414] "containerMap: removing container" podUID="24f2ebc0-b4ff-4412-9bff-5a290e172884" containerName="revoke-keys" Oct 31 01:20:45.296357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296282 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f2ebc0-b4ff-4412-9bff-5a290e172884" containerName="revoke-keys" Oct 31 01:20:45.296582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296453 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f2ebc0-b4ff-4412-9bff-5a290e172884" containerName="revoke-keys" Oct 31 01:20:45.296582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296475 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.296582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.296493 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d640c7-caa1-4ec8-ba7a-d66bf55a69b0" containerName="istio-proxy" Oct 31 01:20:45.301936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.301913 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.328709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.328678 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:45.328917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.328867 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:45.335733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.335704 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t"] Oct 31 01:20:45.342917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.342880 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj"] Oct 31 01:20:45.352355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.352327 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.352476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.352365 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.352901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.352872 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8nj\" (UniqueName: \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.369967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.369940 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj"] Oct 31 01:20:45.370120 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.370042 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.454542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454506 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.454542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454546 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.454815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454574 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8nj\" (UniqueName: \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.454815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454661 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.454815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454700 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.455003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.454939 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lsd\" (UniqueName: \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.455429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.455406 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.456999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.456979 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.478309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.478285 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8nj\" (UniqueName: \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\") pod \"maas-api-revoke-sub-ad1210f6-62n6t\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.556440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.556356 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.556440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.556413 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.556663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.556526 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lsd\" (UniqueName: \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.557036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.557004 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.558905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.558876 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.579919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.579893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lsd\" (UniqueName: \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\") pod \"maas-api-revoke-sub-dd7ad1f2-pqwdj\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.618675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.618643 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:45.686177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.685766 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:45.932779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:45.932751 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t"] Oct 31 01:20:45.933632 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:45.933597 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5925f8_ccaf_4dc3_b5c8_ee3426a257eb.slice/crio-58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4 WatchSource:0}: Error finding container 58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4: Status 404 returned error can't find the container with id 58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4 Oct 31 01:20:46.016126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:46.016096 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj"] Oct 31 01:20:46.016901 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:20:46.016873 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a09f0b_ba6a_41b9_b668_010f56e8af44.slice/crio-cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9 WatchSource:0}: Error finding container cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9: Status 404 returned error can't find the container with id cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9 Oct 31 01:20:46.148304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:46.148266 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" event={"ID":"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb","Type":"ContainerStarted","Data":"58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4"} Oct 31 01:20:46.149869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:46.149845 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" event={"ID":"30a09f0b-ba6a-41b9-b668-010f56e8af44","Type":"ContainerStarted","Data":"cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9"} Oct 31 01:20:47.161771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:47.161735 5011 generic.go:356] "Generic (PLEG): container finished" podID="0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" containerID="1a4abfb0f734f4378551bbea5f90bca66f00f5fc520620a97625d8b946e03fd3" exitCode=0 Oct 31 01:20:47.162242 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:47.161813 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" event={"ID":"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb","Type":"ContainerDied","Data":"1a4abfb0f734f4378551bbea5f90bca66f00f5fc520620a97625d8b946e03fd3"} Oct 31 01:20:47.163960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:47.163936 5011 generic.go:356] "Generic (PLEG): container finished" podID="30a09f0b-ba6a-41b9-b668-010f56e8af44" containerID="dad455066f7d02b7f96fafc2f6f21b8eeb228e03d66f961127a0e8cc946fd442" exitCode=0 Oct 31 01:20:47.164083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:47.164018 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" event={"ID":"30a09f0b-ba6a-41b9-b668-010f56e8af44","Type":"ContainerDied","Data":"dad455066f7d02b7f96fafc2f6f21b8eeb228e03d66f961127a0e8cc946fd442"} Oct 31 01:20:48.338000 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.337964 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:48.389637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.389602 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:48.390405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.390379 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\") pod \"30a09f0b-ba6a-41b9-b668-010f56e8af44\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " Oct 31 01:20:48.390587 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.390569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\" (UniqueName: \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\") pod \"30a09f0b-ba6a-41b9-b668-010f56e8af44\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " Oct 31 01:20:48.390697 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.390615 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\") pod \"30a09f0b-ba6a-41b9-b668-010f56e8af44\" (UID: \"30a09f0b-ba6a-41b9-b668-010f56e8af44\") " Oct 31 01:20:48.390816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.390780 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca" pod "30a09f0b-ba6a-41b9-b668-010f56e8af44" (UID: "30a09f0b-ba6a-41b9-b668-010f56e8af44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:48.391250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.391225 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:48.393250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.393199 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd" pod "30a09f0b-ba6a-41b9-b668-010f56e8af44" (UID: "30a09f0b-ba6a-41b9-b668-010f56e8af44"). InnerVolumeSpecName "kube-api-access-b6lsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:48.393532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.393507 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca" pod "30a09f0b-ba6a-41b9-b668-010f56e8af44" (UID: "30a09f0b-ba6a-41b9-b668-010f56e8af44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:48.492726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.492638 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\") pod \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " Oct 31 01:20:48.492899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.492745 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\") pod \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " Oct 31 01:20:48.492899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.492779 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\" (UniqueName: \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\") pod \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\" (UID: \"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb\") " Oct 31 01:20:48.493083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.493056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca" pod "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" (UID: "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:48.493402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.493382 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lsd\" (UniqueName: \"kubernetes.io/projected/30a09f0b-ba6a-41b9-b668-010f56e8af44-kube-api-access-b6lsd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:48.493498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.493404 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a09f0b-ba6a-41b9-b668-010f56e8af44-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:48.493498 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.493419 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:48.494891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.494869 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca" pod "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" (UID: "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:48.494965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.494943 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj" pod "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" (UID: "0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb"). InnerVolumeSpecName "kube-api-access-zd8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:48.594598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.594569 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:48.594598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:48.594595 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd8nj\" (UniqueName: \"kubernetes.io/projected/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb-kube-api-access-zd8nj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:49.185006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.184975 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" Oct 31 01:20:49.185184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.184982 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t" event={"ID":"0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb","Type":"ContainerDied","Data":"58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4"} Oct 31 01:20:49.185184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.185080 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c5326db9f035b876166c10027df953e72867285e4d424611b7bb11e5e684c4" Oct 31 01:20:49.186700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.186677 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" Oct 31 01:20:49.186700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.186690 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj" event={"ID":"30a09f0b-ba6a-41b9-b668-010f56e8af44","Type":"ContainerDied","Data":"cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9"} Oct 31 01:20:49.186907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:49.186733 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff2dce7c8da524141b0e7c5d5cec1e2fd05a065326995479947e89b6ca7f4d9" Oct 31 01:20:50.742601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:50.742554 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:20:50.743107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:50.742804 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" podUID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" containerName="payload-pre-processing" containerID="cri-o://948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7" gracePeriod=30 Oct 31 01:20:51.141230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.134411 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:20:51.210574 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.210531 5011 generic.go:356] "Generic (PLEG): container finished" podID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" containerID="948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7" exitCode=0 Oct 31 01:20:51.210738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.210584 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" Oct 31 01:20:51.210738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.210620 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" event={"ID":"34c11b71-ec2d-45e1-b9bd-35a92945fd99","Type":"ContainerDied","Data":"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7"} Oct 31 01:20:51.210738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.210685 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb" event={"ID":"34c11b71-ec2d-45e1-b9bd-35a92945fd99","Type":"ContainerDied","Data":"a6780fab36aa28fc6e322ac1d6750a1962ec1ce323e53ef949c679de87028b74"} Oct 31 01:20:51.210738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.210719 5011 scope.go:122] "RemoveContainer" containerID="948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7" Oct 31 01:20:51.225010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.224987 5011 scope.go:122] "RemoveContainer" containerID="948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7" Oct 31 01:20:51.225309 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:20:51.225285 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7\": container with ID starting with 948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7 not found: ID does not exist" containerID="948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7" Oct 31 01:20:51.225389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.225321 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7"} err="failed to get container status \"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7\": rpc error: code = NotFound desc = could not find container \"948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7\": container with ID starting with 948594f09c70e76c9a20d721bad93d98f9f58f2bae8b8d21c944f6de8df150c7 not found: ID does not exist" Oct 31 01:20:51.236700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.236676 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\") pod \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " Oct 31 01:20:51.236830 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.236739 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\" (UniqueName: \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\") pod \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\" (UID: \"34c11b71-ec2d-45e1-b9bd-35a92945fd99\") " Oct 31 01:20:51.237031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.237010 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume" pod "34c11b71-ec2d-45e1-b9bd-35a92945fd99" (UID: "34c11b71-ec2d-45e1-b9bd-35a92945fd99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:51.237189 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.237170 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c11b71-ec2d-45e1-b9bd-35a92945fd99-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:51.238773 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.238747 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j" pod "34c11b71-ec2d-45e1-b9bd-35a92945fd99" (UID: "34c11b71-ec2d-45e1-b9bd-35a92945fd99"). InnerVolumeSpecName "kube-api-access-4f82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:51.283954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.283926 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj"] Oct 31 01:20:51.287473 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.287454 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8d251f-dv8rj"] Oct 31 01:20:51.338305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.338266 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f82j\" (UniqueName: \"kubernetes.io/projected/34c11b71-ec2d-45e1-b9bd-35a92945fd99-kube-api-access-4f82j\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:20:51.560881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.560852 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:20:51.572870 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.572840 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-7847b9ff87-fddcb"] Oct 31 01:20:51.685968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.685894 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" path="/var/lib/kubelet/pods/34c11b71-ec2d-45e1-b9bd-35a92945fd99/volumes" Oct 31 01:20:51.686534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:51.686514 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f795ced0-c7fa-4574-a080-2f681257390f" path="/var/lib/kubelet/pods/f795ced0-c7fa-4574-a080-2f681257390f/volumes" Oct 31 01:20:55.065628 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:55.065592 5011 scope.go:122] "RemoveContainer" containerID="c45f9ed6b1e34b509a1e75ccc529623565010af249d1ab5c88ccf8bd6568eff9" Oct 31 01:20:55.076522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:55.076501 5011 scope.go:122] "RemoveContainer" containerID="d23655d1c20e6c285791ebb487844c1e57ec29c2095db7624a5b636de5be7b2f" Oct 31 01:20:58.035730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:58.035694 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t"] Oct 31 01:20:58.047372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:58.047334 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8"] Oct 31 01:20:58.050883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:58.050855 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab1ca71-fsp9t"] Oct 31 01:20:58.053780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:58.053758 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6908e8-brjp8"] Oct 31 01:20:59.683034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.682996 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0d58b0-12c4-4991-89c2-2b867869829f" path="/var/lib/kubelet/pods/ba0d58b0-12c4-4991-89c2-2b867869829f/volumes" Oct 31 01:20:59.683491 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.683476 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d568a9ce-45ab-4ab7-b4b1-88543bb929fd" path="/var/lib/kubelet/pods/d568a9ce-45ab-4ab7-b4b1-88543bb929fd/volumes" Oct 31 01:20:59.809385 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809350 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5"] Oct 31 01:20:59.809878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809865 5011 cpu_manager.go:414] "containerMap: removing container" podUID="30a09f0b-ba6a-41b9-b668-010f56e8af44" containerName="revoke-keys" Oct 31 01:20:59.809922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809879 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a09f0b-ba6a-41b9-b668-010f56e8af44" containerName="revoke-keys" Oct 31 01:20:59.809922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809918 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" containerName="revoke-keys" Oct 31 01:20:59.809992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809923 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" containerName="revoke-keys" Oct 31 01:20:59.809992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809939 5011 cpu_manager.go:414] "containerMap: removing container" podUID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" containerName="payload-pre-processing" Oct 31 01:20:59.809992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.809945 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" containerName="payload-pre-processing" Oct 31 01:20:59.810081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.810024 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" containerName="revoke-keys" Oct 31 01:20:59.810081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.810034 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c11b71-ec2d-45e1-b9bd-35a92945fd99" containerName="payload-pre-processing" Oct 31 01:20:59.810081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.810041 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a09f0b-ba6a-41b9-b668-010f56e8af44" containerName="revoke-keys" Oct 31 01:20:59.814534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.814516 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:20:59.834741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.834710 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5"] Oct 31 01:20:59.875848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.875817 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:59.876012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.875901 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:59.954456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.954369 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzt7z\" (UniqueName: \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:20:59.954456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.954416 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:20:59.954696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:20:59.954649 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.056321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.056286 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.056512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.056385 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzt7z\" (UniqueName: \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.056512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.056412 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.057063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.057041 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.058944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.058913 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.080068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.080043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzt7z\" (UniqueName: \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.129894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.129863 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:00.445103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:00.444966 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5"] Oct 31 01:21:01.327185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:01.327143 5011 generic.go:356] "Generic (PLEG): container finished" podID="814dfdc9-931b-4908-883b-a9ad63e9d979" containerID="e0e0eddf692f9e0f714590ee305baddeabc5aff83d84fa9617b3491c293d5a19" exitCode=0 Oct 31 01:21:01.327664 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:01.327242 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" event={"ID":"814dfdc9-931b-4908-883b-a9ad63e9d979","Type":"ContainerDied","Data":"e0e0eddf692f9e0f714590ee305baddeabc5aff83d84fa9617b3491c293d5a19"} Oct 31 01:21:01.327664 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:01.327333 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" event={"ID":"814dfdc9-931b-4908-883b-a9ad63e9d979","Type":"ContainerStarted","Data":"a617f51dc9738b762121a70540dafa684a0bc2eb3201acb017297d3c5662aade"} Oct 31 01:21:01.950356 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:01.950321 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:21:02.574986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.574952 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:02.698742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.698639 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\") pod \"814dfdc9-931b-4908-883b-a9ad63e9d979\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " Oct 31 01:21:02.698898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.698862 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\") pod \"814dfdc9-931b-4908-883b-a9ad63e9d979\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " Oct 31 01:21:02.698958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.698918 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\" (UniqueName: \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\") pod \"814dfdc9-931b-4908-883b-a9ad63e9d979\" (UID: \"814dfdc9-931b-4908-883b-a9ad63e9d979\") " Oct 31 01:21:02.699044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.699016 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca" pod "814dfdc9-931b-4908-883b-a9ad63e9d979" (UID: "814dfdc9-931b-4908-883b-a9ad63e9d979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:02.699281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.699262 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.701102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.701072 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z" pod "814dfdc9-931b-4908-883b-a9ad63e9d979" (UID: "814dfdc9-931b-4908-883b-a9ad63e9d979"). InnerVolumeSpecName "kube-api-access-wzt7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:02.701191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.701133 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca" pod "814dfdc9-931b-4908-883b-a9ad63e9d979" (UID: "814dfdc9-931b-4908-883b-a9ad63e9d979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:02.800409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.800372 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814dfdc9-931b-4908-883b-a9ad63e9d979-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.800409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:02.800410 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzt7z\" (UniqueName: \"kubernetes.io/projected/814dfdc9-931b-4908-883b-a9ad63e9d979-kube-api-access-wzt7z\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:03.251308 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.251272 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc"] Oct 31 01:21:03.251879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.251863 5011 cpu_manager.go:414] "containerMap: removing container" podUID="814dfdc9-931b-4908-883b-a9ad63e9d979" containerName="revoke-keys" Oct 31 01:21:03.251921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.251880 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814dfdc9-931b-4908-883b-a9ad63e9d979" containerName="revoke-keys" Oct 31 01:21:03.251979 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.251970 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="814dfdc9-931b-4908-883b-a9ad63e9d979" containerName="revoke-keys" Oct 31 01:21:03.257010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.256979 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.279133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.279094 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc"] Oct 31 01:21:03.291358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.291324 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp"] Oct 31 01:21:03.296384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.296355 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.296897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.296859 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:21:03.316015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.315978 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp"] Oct 31 01:21:03.378060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.378022 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" Oct 31 01:21:03.378287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.378060 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5" event={"ID":"814dfdc9-931b-4908-883b-a9ad63e9d979","Type":"ContainerDied","Data":"a617f51dc9738b762121a70540dafa684a0bc2eb3201acb017297d3c5662aade"} Oct 31 01:21:03.378287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.378103 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a617f51dc9738b762121a70540dafa684a0bc2eb3201acb017297d3c5662aade" Oct 31 01:21:03.412595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.412559 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.412731 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.412692 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.412784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.412759 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kt6v\" (UniqueName: \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.412896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.412875 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.413092 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.413066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw27b\" (UniqueName: \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.413193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.413170 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.514987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.514897 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw27b\" (UniqueName: \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.514987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.514955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.515245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515002 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.515245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515140 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.515368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515240 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kt6v\" (UniqueName: \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.515368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.515895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515863 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.515895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.515892 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.517709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.517685 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.517798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.517783 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.538923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.538898 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw27b\" (UniqueName: \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\") pod \"maas-api-revoke-sub-ed03ec3a-b68sp\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.539475 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.539447 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kt6v\" (UniqueName: \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\") pod \"maas-api-revoke-sub-52accf06-xcltc\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.584272 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.584242 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:03.626060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.626024 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:03.913644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.913614 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc"] Oct 31 01:21:03.914342 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:03.914310 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899a2659_7305_4596_8013_882673f02051.slice/crio-405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba WatchSource:0}: Error finding container 405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba: Status 404 returned error can't find the container with id 405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba Oct 31 01:21:03.956265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:03.956236 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp"] Oct 31 01:21:03.959584 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:03.959539 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5953c647_7300_4db4_937a_11a980430014.slice/crio-42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d WatchSource:0}: Error finding container 42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d: Status 404 returned error can't find the container with id 42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d Oct 31 01:21:04.388949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:04.388908 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" event={"ID":"899a2659-7305-4596-8013-882673f02051","Type":"ContainerStarted","Data":"405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba"} Oct 31 01:21:04.390728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:04.390663 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" event={"ID":"5953c647-7300-4db4-937a-11a980430014","Type":"ContainerStarted","Data":"42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d"} Oct 31 01:21:05.399766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:05.399727 5011 generic.go:356] "Generic (PLEG): container finished" podID="5953c647-7300-4db4-937a-11a980430014" containerID="9cd005b30ea918464392e81e8184c0666c8bf65c638989698fef588bc5774f5c" exitCode=0 Oct 31 01:21:05.400241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:05.399821 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" event={"ID":"5953c647-7300-4db4-937a-11a980430014","Type":"ContainerDied","Data":"9cd005b30ea918464392e81e8184c0666c8bf65c638989698fef588bc5774f5c"} Oct 31 01:21:05.401753 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:05.401727 5011 generic.go:356] "Generic (PLEG): container finished" podID="899a2659-7305-4596-8013-882673f02051" containerID="290e9a7faf785b9b9519319027587b0a689771fcfdf50b563b2c2858f62da00d" exitCode=0 Oct 31 01:21:05.401875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:05.401768 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" event={"ID":"899a2659-7305-4596-8013-882673f02051","Type":"ContainerDied","Data":"290e9a7faf785b9b9519319027587b0a689771fcfdf50b563b2c2858f62da00d"} Oct 31 01:21:06.613040 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.612887 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:06.634288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.634267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.703769 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\" (UniqueName: \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\") pod \"5953c647-7300-4db4-937a-11a980430014\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.703817 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\" (UniqueName: \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\") pod \"899a2659-7305-4596-8013-882673f02051\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.703868 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\") pod \"899a2659-7305-4596-8013-882673f02051\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.703900 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\") pod \"5953c647-7300-4db4-937a-11a980430014\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.704005 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\") pod \"899a2659-7305-4596-8013-882673f02051\" (UID: \"899a2659-7305-4596-8013-882673f02051\") " Oct 31 01:21:06.707777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.704108 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\") pod \"5953c647-7300-4db4-937a-11a980430014\" (UID: \"5953c647-7300-4db4-937a-11a980430014\") " Oct 31 01:21:06.713568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.713522 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca" pod "899a2659-7305-4596-8013-882673f02051" (UID: "899a2659-7305-4596-8013-882673f02051"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:06.714327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.714300 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca" pod "5953c647-7300-4db4-937a-11a980430014" (UID: "5953c647-7300-4db4-937a-11a980430014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:06.715597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.715573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca" pod "5953c647-7300-4db4-937a-11a980430014" (UID: "5953c647-7300-4db4-937a-11a980430014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:06.715903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.715881 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v" pod "899a2659-7305-4596-8013-882673f02051" (UID: "899a2659-7305-4596-8013-882673f02051"). InnerVolumeSpecName "kube-api-access-7kt6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:06.716016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.715946 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b" pod "5953c647-7300-4db4-937a-11a980430014" (UID: "5953c647-7300-4db4-937a-11a980430014"). InnerVolumeSpecName "kube-api-access-tw27b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:06.716971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.716954 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca" pod "899a2659-7305-4596-8013-882673f02051" (UID: "899a2659-7305-4596-8013-882673f02051"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:06.806410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806370 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953c647-7300-4db4-937a-11a980430014-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:06.806410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806410 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw27b\" (UniqueName: \"kubernetes.io/projected/5953c647-7300-4db4-937a-11a980430014-kube-api-access-tw27b\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:06.806622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806426 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kt6v\" (UniqueName: \"kubernetes.io/projected/899a2659-7305-4596-8013-882673f02051-kube-api-access-7kt6v\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:06.806622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806440 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899a2659-7305-4596-8013-882673f02051-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:06.806622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806453 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953c647-7300-4db4-937a-11a980430014-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:06.806622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:06.806465 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899a2659-7305-4596-8013-882673f02051-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.420772 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" event={"ID":"899a2659-7305-4596-8013-882673f02051","Type":"ContainerDied","Data":"405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba"} Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.420878 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405e288c6cf39e87774c55e4a211c069c83facd634916e932ef2e38213dd16ba" Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.420973 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc" Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.423184 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" event={"ID":"5953c647-7300-4db4-937a-11a980430014","Type":"ContainerDied","Data":"42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d"} Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.423274 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42994067ba9f865e5106cad283c578760a47f26b0b09dc3aab7007880e4e4e7d" Oct 31 01:21:07.427944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:07.423357 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp" Oct 31 01:21:10.280525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:10.280490 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 01:21:12.460135 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460096 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k"] Oct 31 01:21:12.460719 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460702 5011 cpu_manager.go:414] "containerMap: removing container" podUID="899a2659-7305-4596-8013-882673f02051" containerName="revoke-keys" Oct 31 01:21:12.460777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460721 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899a2659-7305-4596-8013-882673f02051" containerName="revoke-keys" Oct 31 01:21:12.460777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460741 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5953c647-7300-4db4-937a-11a980430014" containerName="revoke-keys" Oct 31 01:21:12.460777 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460746 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5953c647-7300-4db4-937a-11a980430014" containerName="revoke-keys" Oct 31 01:21:12.460913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460847 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="899a2659-7305-4596-8013-882673f02051" containerName="revoke-keys" Oct 31 01:21:12.460913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.460860 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5953c647-7300-4db4-937a-11a980430014" containerName="revoke-keys" Oct 31 01:21:12.465710 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.465690 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.481466 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.481437 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k"] Oct 31 01:21:12.488541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.488518 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:12.488940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.488922 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:21:12.598779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.598741 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.598920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.598796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnk4\" (UniqueName: \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.599036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.599011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.701378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.700972 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.701378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.701081 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.701378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.701114 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnk4\" (UniqueName: \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.704126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.704095 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.708720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.708677 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.725079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.725008 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnk4\" (UniqueName: \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\") pod \"maas-api-revoke-sub-00903595-b5h6k\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:12.781016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:12.780992 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:13.089510 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:13.089469 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4933f508_43ea_413b_99a3_53f433d53e60.slice/crio-f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1 WatchSource:0}: Error finding container f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1: Status 404 returned error can't find the container with id f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1 Oct 31 01:21:13.091556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:13.091526 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k"] Oct 31 01:21:13.475255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:13.475214 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" event={"ID":"4933f508-43ea-413b-99a3-53f433d53e60","Type":"ContainerStarted","Data":"bf6c76c236162a617d4cab5692ea289c019143fd6ffdbac76ea1a0021acb91b1"} Oct 31 01:21:13.475599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:13.475278 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" event={"ID":"4933f508-43ea-413b-99a3-53f433d53e60","Type":"ContainerStarted","Data":"f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1"} Oct 31 01:21:13.522141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:13.522093 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" podStartSLOduration=1.52207919 podStartE2EDuration="1.52207919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:13.521924831 +0000 UTC m=+3741.971012470" watchObservedRunningTime="2026-10-31 01:21:13.52207919 +0000 UTC m=+3741.971166862" Oct 31 01:21:14.519077 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:14.519030 5011 generic.go:356] "Generic (PLEG): container finished" podID="4933f508-43ea-413b-99a3-53f433d53e60" containerID="bf6c76c236162a617d4cab5692ea289c019143fd6ffdbac76ea1a0021acb91b1" exitCode=0 Oct 31 01:21:14.519611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:14.519260 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" event={"ID":"4933f508-43ea-413b-99a3-53f433d53e60","Type":"ContainerDied","Data":"bf6c76c236162a617d4cab5692ea289c019143fd6ffdbac76ea1a0021acb91b1"} Oct 31 01:21:14.701533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:14.701380 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt"] Oct 31 01:21:14.705605 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:14.705572 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4207cd-8sdxt"] Oct 31 01:21:15.281218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.281165 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" containerID="cri-o://770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64" gracePeriod=25 Oct 31 01:21:15.684332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.684305 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:15.684692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.684663 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f371aa21-b210-4b6e-8f94-e19620f301c3" path="/var/lib/kubelet/pods/f371aa21-b210-4b6e-8f94-e19620f301c3/volumes" Oct 31 01:21:15.844724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.844631 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\" (UniqueName: \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\") pod \"4933f508-43ea-413b-99a3-53f433d53e60\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " Oct 31 01:21:15.844916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.844780 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\") pod \"4933f508-43ea-413b-99a3-53f433d53e60\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " Oct 31 01:21:15.844916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.844861 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\") pod \"4933f508-43ea-413b-99a3-53f433d53e60\" (UID: \"4933f508-43ea-413b-99a3-53f433d53e60\") " Oct 31 01:21:15.845335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.845302 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca" pod "4933f508-43ea-413b-99a3-53f433d53e60" (UID: "4933f508-43ea-413b-99a3-53f433d53e60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:15.845526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.845484 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:15.847116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.847079 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4" pod "4933f508-43ea-413b-99a3-53f433d53e60" (UID: "4933f508-43ea-413b-99a3-53f433d53e60"). InnerVolumeSpecName "kube-api-access-9lnk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:15.847396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.847368 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca" pod "4933f508-43ea-413b-99a3-53f433d53e60" (UID: "4933f508-43ea-413b-99a3-53f433d53e60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:15.946298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.946267 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4933f508-43ea-413b-99a3-53f433d53e60-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:15.946298 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:15.946297 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnk4\" (UniqueName: \"kubernetes.io/projected/4933f508-43ea-413b-99a3-53f433d53e60-kube-api-access-9lnk4\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:16.539028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:16.538996 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" Oct 31 01:21:16.539028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:16.539016 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k" event={"ID":"4933f508-43ea-413b-99a3-53f433d53e60","Type":"ContainerDied","Data":"f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1"} Oct 31 01:21:16.539315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:16.539089 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ffc7391e2ceff6c7bc051fd2b56d71ac5a7f4ae3ac8633072bf1533de65ba1" Oct 31 01:21:18.416230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:18.415636 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:21:18.766649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:18.766568 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h"] Oct 31 01:21:18.777923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:18.777893 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b024b5-hwh5h"] Oct 31 01:21:19.683070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:19.683035 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eadde956-9fe6-48ff-8fe4-5ec94c580cc1" path="/var/lib/kubelet/pods/eadde956-9fe6-48ff-8fe4-5ec94c580cc1/volumes" Oct 31 01:21:20.172137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:20.172101 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:21:22.951434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.951399 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj"] Oct 31 01:21:22.951947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.951934 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4933f508-43ea-413b-99a3-53f433d53e60" containerName="revoke-keys" Oct 31 01:21:22.951999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.951949 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4933f508-43ea-413b-99a3-53f433d53e60" containerName="revoke-keys" Oct 31 01:21:22.952074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.952063 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4933f508-43ea-413b-99a3-53f433d53e60" containerName="revoke-keys" Oct 31 01:21:22.957037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.957018 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:22.990003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:22.989981 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:23.002185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.002162 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:21:23.008661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.008633 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj"] Oct 31 01:21:23.048942 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.048907 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.049087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.049062 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.049291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.049266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbsmd\" (UniqueName: \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.157801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.157753 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.157998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.157934 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.158169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.158142 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbsmd\" (UniqueName: \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.159917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.159070 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.172290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.172133 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.189173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.189141 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbsmd\" (UniqueName: \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\") pod \"maas-api-revoke-sub-cd817dd4-fhdpj\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.271614 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.271531 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:23.578542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.578508 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj"] Oct 31 01:21:23.579307 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:23.579275 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a0bf4d_57c4_47b9_9bb7_5fc3eace48f5.slice/crio-b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579 WatchSource:0}: Error finding container b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579: Status 404 returned error can't find the container with id b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579 Oct 31 01:21:23.599680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:23.599646 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" event={"ID":"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5","Type":"ContainerStarted","Data":"b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579"} Oct 31 01:21:24.561913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:24.561875 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5"] Oct 31 01:21:24.573408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:24.573379 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-bcdb4e-qhgg5"] Oct 31 01:21:24.629924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:24.629882 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" event={"ID":"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5","Type":"ContainerStarted","Data":"9f3a73cc501880e93e7186ed7bbdc76ba8ef822665935da050ceb5ff24bb40d5"} Oct 31 01:21:24.673374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:24.673317 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" podStartSLOduration=2.67330322 podStartE2EDuration="2.67330322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:24.670285767 +0000 UTC m=+3753.119373405" watchObservedRunningTime="2026-10-31 01:21:24.67330322 +0000 UTC m=+3753.122390858" Oct 31 01:21:25.650794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:25.650754 5011 generic.go:356] "Generic (PLEG): container finished" podID="81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" containerID="9f3a73cc501880e93e7186ed7bbdc76ba8ef822665935da050ceb5ff24bb40d5" exitCode=0 Oct 31 01:21:25.651201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:25.650836 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" event={"ID":"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5","Type":"ContainerDied","Data":"9f3a73cc501880e93e7186ed7bbdc76ba8ef822665935da050ceb5ff24bb40d5"} Oct 31 01:21:25.682971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:25.682941 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814dfdc9-931b-4908-883b-a9ad63e9d979" path="/var/lib/kubelet/pods/814dfdc9-931b-4908-883b-a9ad63e9d979/volumes" Oct 31 01:21:26.812324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.812295 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:26.936002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.935899 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\" (UniqueName: \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\") pod \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " Oct 31 01:21:26.936184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.936057 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\") pod \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " Oct 31 01:21:26.936184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.936120 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\") pod \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\" (UID: \"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5\") " Oct 31 01:21:26.936486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.936458 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca" pod "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" (UID: "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:26.936700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.936668 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:26.938440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.938413 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd" pod "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" (UID: "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5"). InnerVolumeSpecName "kube-api-access-lbsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:26.938731 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:26.938713 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca" pod "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" (UID: "81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:27.038515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:27.038472 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbsmd\" (UniqueName: \"kubernetes.io/projected/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-kube-api-access-lbsmd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:27.038515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:27.038506 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:27.668396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:27.668363 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" Oct 31 01:21:27.668572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:27.668370 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj" event={"ID":"81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5","Type":"ContainerDied","Data":"b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579"} Oct 31 01:21:27.668572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:27.668477 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3713dd91d7b93de646eec5e4da395cc88d5c26140123c040b3c4a1d14e87579" Oct 31 01:21:28.377256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:28.377217 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:21:30.110389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.110353 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 01:21:30.111103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.111067 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" podUID="74627e15-7c07-4e25-bca2-6edd558d685e" containerName="maas-api" containerID="cri-o://d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a" gracePeriod=30 Oct 31 01:21:30.482600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.481703 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 01:21:30.625583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.625542 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\" (UniqueName: \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\") pod \"74627e15-7c07-4e25-bca2-6edd558d685e\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " Oct 31 01:21:30.625883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.625854 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\") pod \"74627e15-7c07-4e25-bca2-6edd558d685e\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " Oct 31 01:21:30.625986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.625954 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\") pod \"74627e15-7c07-4e25-bca2-6edd558d685e\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " Oct 31 01:21:30.626285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.626259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\") pod \"74627e15-7c07-4e25-bca2-6edd558d685e\" (UID: \"74627e15-7c07-4e25-bca2-6edd558d685e\") " Oct 31 01:21:30.626720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.626673 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca" pod "74627e15-7c07-4e25-bca2-6edd558d685e" (UID: "74627e15-7c07-4e25-bca2-6edd558d685e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:30.627456 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.627433 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74627e15-7c07-4e25-bca2-6edd558d685e-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:30.634540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.633124 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls" pod "74627e15-7c07-4e25-bca2-6edd558d685e" (UID: "74627e15-7c07-4e25-bca2-6edd558d685e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:30.634828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.634776 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx" pod "74627e15-7c07-4e25-bca2-6edd558d685e" (UID: "74627e15-7c07-4e25-bca2-6edd558d685e"). InnerVolumeSpecName "kube-api-access-dtpvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:30.634924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.634882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs" pod "74627e15-7c07-4e25-bca2-6edd558d685e" (UID: "74627e15-7c07-4e25-bca2-6edd558d685e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:30.693417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.693388 5011 generic.go:356] "Generic (PLEG): container finished" podID="74627e15-7c07-4e25-bca2-6edd558d685e" containerID="d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a" exitCode=0 Oct 31 01:21:30.693582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.693431 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" Oct 31 01:21:30.693582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.693467 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" event={"ID":"74627e15-7c07-4e25-bca2-6edd558d685e","Type":"ContainerDied","Data":"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a"} Oct 31 01:21:30.693582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.693521 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw" event={"ID":"74627e15-7c07-4e25-bca2-6edd558d685e","Type":"ContainerDied","Data":"a7fbdc63561f88b6b00ebc74c3ae901d0cda71430645361906145c37afb53045"} Oct 31 01:21:30.693582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.693551 5011 scope.go:122] "RemoveContainer" containerID="d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a" Oct 31 01:21:30.705667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.705646 5011 scope.go:122] "RemoveContainer" containerID="d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a" Oct 31 01:21:30.705960 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:30.705940 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a\": container with ID starting with d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a not found: ID does not exist" containerID="d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a" Oct 31 01:21:30.706035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.705969 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a"} err="failed to get container status \"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a\": rpc error: code = NotFound desc = could not find container \"d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a\": container with ID starting with d4067c10e4e9bded62e46573fdad3569f65729eccdf77f390ec957b6c4945f6a not found: ID does not exist" Oct 31 01:21:30.728760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.728728 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpvx\" (UniqueName: \"kubernetes.io/projected/74627e15-7c07-4e25-bca2-6edd558d685e-kube-api-access-dtpvx\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:30.728760 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.728758 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:30.728923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.728774 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74627e15-7c07-4e25-bca2-6edd558d685e-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:30.757193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.757164 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 01:21:30.763165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.763138 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-671959-6d88b66cd-28lpw"] Oct 31 01:21:30.861567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.861528 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst"] Oct 31 01:21:30.865007 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:30.864971 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-671959-xfcst"] Oct 31 01:21:31.683991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:31.683956 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f2ebc0-b4ff-4412-9bff-5a290e172884" path="/var/lib/kubelet/pods/24f2ebc0-b4ff-4412-9bff-5a290e172884/volumes" Oct 31 01:21:31.684438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:31.684421 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74627e15-7c07-4e25-bca2-6edd558d685e" path="/var/lib/kubelet/pods/74627e15-7c07-4e25-bca2-6edd558d685e/volumes" Oct 31 01:21:31.957890 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:31.957793 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:21:32.874626 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.874591 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb"] Oct 31 01:21:32.875380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875355 5011 cpu_manager.go:414] "containerMap: removing container" podUID="81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" containerName="revoke-keys" Oct 31 01:21:32.875380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875380 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" containerName="revoke-keys" Oct 31 01:21:32.875572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875406 5011 cpu_manager.go:414] "containerMap: removing container" podUID="74627e15-7c07-4e25-bca2-6edd558d685e" containerName="maas-api" Oct 31 01:21:32.875572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875413 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74627e15-7c07-4e25-bca2-6edd558d685e" containerName="maas-api" Oct 31 01:21:32.875677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875585 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" containerName="revoke-keys" Oct 31 01:21:32.875677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.875603 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="74627e15-7c07-4e25-bca2-6edd558d685e" containerName="maas-api" Oct 31 01:21:32.880566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.880546 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:32.896956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.896928 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb"] Oct 31 01:21:32.897272 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.897248 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:32.897441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.897417 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:21:32.954508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.954468 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:32.954689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.954534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:32.954689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:32.954649 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgrc\" (UniqueName: \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.055893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.055848 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.056127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.055922 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.056127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.055996 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgrc\" (UniqueName: \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.056707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.056682 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.058361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.058339 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.079724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.079698 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgrc\" (UniqueName: \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.196059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.195983 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:33.511291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.511255 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb"] Oct 31 01:21:33.512449 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:33.512423 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5137c3f0_e155_4d4c_bf53_5cf18984b3dc.slice/crio-e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93 WatchSource:0}: Error finding container e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93: Status 404 returned error can't find the container with id e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93 Oct 31 01:21:33.723021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.722980 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" event={"ID":"5137c3f0-e155-4d4c-bf53-5cf18984b3dc","Type":"ContainerStarted","Data":"e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93"} Oct 31 01:21:33.962234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:33.960888 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:21:34.736792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:34.736761 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" event={"ID":"5137c3f0-e155-4d4c-bf53-5cf18984b3dc","Type":"ContainerStarted","Data":"cba557e711264cbd549ebdafe9190bc6b71c0e350106b2a2037c9a2518ce19b3"} Oct 31 01:21:35.193416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.193379 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerName="main" containerID="cri-o://658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de" gracePeriod=45 Oct 31 01:21:35.197559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.197525 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": EOF" Oct 31 01:21:35.597306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.597269 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:21:35.613562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.613533 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 01:21:35.696501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696465 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\") pod \"f39ac115-645a-4611-8d44-d4aea5b4b13c\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " Oct 31 01:21:35.696686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696529 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\" (UniqueName: \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.696686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696591 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.696686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696629 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\" (UniqueName: \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\") pod \"f39ac115-645a-4611-8d44-d4aea5b4b13c\" (UID: \"f39ac115-645a-4611-8d44-d4aea5b4b13c\") " Oct 31 01:21:35.696686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696659 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.696937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696766 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.696937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696853 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.696937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696863 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:35.696937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696899 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume" pod "f39ac115-645a-4611-8d44-d4aea5b4b13c" (UID: "f39ac115-645a-4611-8d44-d4aea5b4b13c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:35.697157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.696948 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\") pod \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\" (UID: \"d72c884c-40e1-45d5-bb7a-6cf677d95b8e\") " Oct 31 01:21:35.697276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.697247 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:35.697474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.697445 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:35.697586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.697536 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39ac115-645a-4611-8d44-d4aea5b4b13c-plugins-config-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.697586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.697560 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.697586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.697576 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.700953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.700915 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:35.701184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.701164 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:35.701311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.701232 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k" pod "d72c884c-40e1-45d5-bb7a-6cf677d95b8e" (UID: "d72c884c-40e1-45d5-bb7a-6cf677d95b8e"). InnerVolumeSpecName "kube-api-access-4hn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:35.701689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.701665 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88" pod "f39ac115-645a-4611-8d44-d4aea5b4b13c" (UID: "f39ac115-645a-4611-8d44-d4aea5b4b13c"). InnerVolumeSpecName "kube-api-access-w2z88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:35.757260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.755783 5011 generic.go:356] "Generic (PLEG): container finished" podID="5137c3f0-e155-4d4c-bf53-5cf18984b3dc" containerID="cba557e711264cbd549ebdafe9190bc6b71c0e350106b2a2037c9a2518ce19b3" exitCode=0 Oct 31 01:21:35.757260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.756008 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" event={"ID":"5137c3f0-e155-4d4c-bf53-5cf18984b3dc","Type":"ContainerDied","Data":"cba557e711264cbd549ebdafe9190bc6b71c0e350106b2a2037c9a2518ce19b3"} Oct 31 01:21:35.760240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.758506 5011 generic.go:356] "Generic (PLEG): container finished" podID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerID="658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de" exitCode=0 Oct 31 01:21:35.760240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.758552 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" Oct 31 01:21:35.760240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.758596 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" event={"ID":"d72c884c-40e1-45d5-bb7a-6cf677d95b8e","Type":"ContainerDied","Data":"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de"} Oct 31 01:21:35.760240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.758635 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb" event={"ID":"d72c884c-40e1-45d5-bb7a-6cf677d95b8e","Type":"ContainerDied","Data":"fe27477788d432be4eec0af5705b67eedab1c3746074ad159f640e87e7652012"} Oct 31 01:21:35.760240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.758658 5011 scope.go:122] "RemoveContainer" containerID="658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de" Oct 31 01:21:35.767245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.760600 5011 generic.go:356] "Generic (PLEG): container finished" podID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerID="770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64" exitCode=0 Oct 31 01:21:35.767245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.760716 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" event={"ID":"f39ac115-645a-4611-8d44-d4aea5b4b13c","Type":"ContainerDied","Data":"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64"} Oct 31 01:21:35.767245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.760748 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" event={"ID":"f39ac115-645a-4611-8d44-d4aea5b4b13c","Type":"ContainerDied","Data":"d7afa62510156e80e5b6c751211ba9cde09651a58bab89f8ea0573cfcbf7af18"} Oct 31 01:21:35.767245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.760798 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh" Oct 31 01:21:35.772733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.772716 5011 scope.go:122] "RemoveContainer" containerID="658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de" Oct 31 01:21:35.773233 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:35.773170 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de\": container with ID starting with 658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de not found: ID does not exist" containerID="658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de" Oct 31 01:21:35.773289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.773239 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de"} err="failed to get container status \"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de\": rpc error: code = NotFound desc = could not find container \"658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de\": container with ID starting with 658102e42660c31aaf28e72b5f6fda7233888a72fc31996a4d4a60b6810fc8de not found: ID does not exist" Oct 31 01:21:35.773289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.773262 5011 scope.go:122] "RemoveContainer" containerID="770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64" Oct 31 01:21:35.784139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.784121 5011 scope.go:122] "RemoveContainer" containerID="770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64" Oct 31 01:21:35.784470 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:35.784452 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64\": container with ID starting with 770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64 not found: ID does not exist" containerID="770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64" Oct 31 01:21:35.784530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.784476 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64"} err="failed to get container status \"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64\": rpc error: code = NotFound desc = could not find container \"770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64\": container with ID starting with 770a2350f7f1b38f4bca49bf344d5ba293093cf5e5795325ebd9fb3f1d348f64 not found: ID does not exist" Oct 31 01:21:35.799289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.799260 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.799289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.799290 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.799446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.799301 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hn9k\" (UniqueName: \"kubernetes.io/projected/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-kube-api-access-4hn9k\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.799446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.799312 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2z88\" (UniqueName: \"kubernetes.io/projected/f39ac115-645a-4611-8d44-d4aea5b4b13c-kube-api-access-w2z88\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.799446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.799320 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72c884c-40e1-45d5-bb7a-6cf677d95b8e-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:35.813130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.813079 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 01:21:35.817062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.817037 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-bcdb4e-7db995c565-6fvlh"] Oct 31 01:21:35.851763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.851394 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:21:35.861405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.861370 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4207cd/ipp-route-4207cd-kserve-6b48c4cb45-7w4rb"] Oct 31 01:21:35.904861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:35.904838 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:36.000866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.000794 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\" (UniqueName: \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\") pod \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " Oct 31 01:21:36.001021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.000928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\") pod \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " Oct 31 01:21:36.001021 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.000972 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\") pod \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\" (UID: \"5137c3f0-e155-4d4c-bf53-5cf18984b3dc\") " Oct 31 01:21:36.001644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.001425 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca" pod "5137c3f0-e155-4d4c-bf53-5cf18984b3dc" (UID: "5137c3f0-e155-4d4c-bf53-5cf18984b3dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:36.002996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.002971 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca" pod "5137c3f0-e155-4d4c-bf53-5cf18984b3dc" (UID: "5137c3f0-e155-4d4c-bf53-5cf18984b3dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:36.002996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.002982 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc" pod "5137c3f0-e155-4d4c-bf53-5cf18984b3dc" (UID: "5137c3f0-e155-4d4c-bf53-5cf18984b3dc"). InnerVolumeSpecName "kube-api-access-dxgrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:36.102998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.102957 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgrc\" (UniqueName: \"kubernetes.io/projected/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-kube-api-access-dxgrc\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:36.102998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.102999 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:36.103237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.103014 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137c3f0-e155-4d4c-bf53-5cf18984b3dc-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:36.769674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.769642 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" Oct 31 01:21:36.769674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.769663 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb" event={"ID":"5137c3f0-e155-4d4c-bf53-5cf18984b3dc","Type":"ContainerDied","Data":"e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93"} Oct 31 01:21:36.770323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:36.769706 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98db57916af140716a2ec648cbc8b80de2b80789d66a7583c1160df068f7f93" Oct 31 01:21:37.684178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:37.684140 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" path="/var/lib/kubelet/pods/d72c884c-40e1-45d5-bb7a-6cf677d95b8e/volumes" Oct 31 01:21:37.684725 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:37.684698 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" path="/var/lib/kubelet/pods/f39ac115-645a-4611-8d44-d4aea5b4b13c/volumes" Oct 31 01:21:37.768306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:37.764531 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 01:21:37.768306 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:37.764767 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" containerID="cri-o://519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a" gracePeriod=30 Oct 31 01:21:41.228247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:41.228173 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:21:41.228693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:41.228292 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:21:43.051163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.051105 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 01:21:43.088344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088318 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.088472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088355 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.088472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088391 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.088472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088453 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.088594 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088542 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088758 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088792 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088821 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\") pod \"d0c1d831-fefc-4d9e-9fc9-107235d04648\" (UID: \"d0c1d831-fefc-4d9e-9fc9-107235d04648\") " Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088815 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.088959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:43.089202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089026 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:43.089691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089154 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:43.089691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089524 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.089691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089547 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.089691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089562 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.089691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089574 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c1d831-fefc-4d9e-9fc9-107235d04648-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.089928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.089906 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:43.090904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.090872 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "kube-api-access-2jwmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:43.091003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.090963 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:43.091336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.091312 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:43.091442 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.091337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token" pod "d0c1d831-fefc-4d9e-9fc9-107235d04648" (UID: "d0c1d831-fefc-4d9e-9fc9-107235d04648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:43.190992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.190958 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.190992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.190987 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.191200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.191003 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwmf\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-kube-api-access-2jwmf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.191200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.191016 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.191200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.191028 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c1d831-fefc-4d9e-9fc9-107235d04648-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:43.826381 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.826346 5011 generic.go:356] "Generic (PLEG): container finished" podID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerID="519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a" exitCode=0 Oct 31 01:21:43.826590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.826392 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" Oct 31 01:21:43.826590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.826435 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" event={"ID":"d0c1d831-fefc-4d9e-9fc9-107235d04648","Type":"ContainerDied","Data":"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a"} Oct 31 01:21:43.826590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.826496 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x" event={"ID":"d0c1d831-fefc-4d9e-9fc9-107235d04648","Type":"ContainerDied","Data":"3d38c00ba420a4d7d80fec14c77a9e9be9a4d0ace36103bc7b4be61625fc8797"} Oct 31 01:21:43.826590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.826527 5011 scope.go:122] "RemoveContainer" containerID="519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a" Oct 31 01:21:43.843584 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.843547 5011 scope.go:122] "RemoveContainer" containerID="519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a" Oct 31 01:21:43.843813 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:43.843790 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a\": container with ID starting with 519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a not found: ID does not exist" containerID="519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a" Oct 31 01:21:43.843878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.843822 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a"} err="failed to get container status \"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a\": rpc error: code = NotFound desc = could not find container \"519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a\": container with ID starting with 519c349ae1a319d326b85ea2032b0dab9f8d97ac37c68d030064254ed6aa770a not found: ID does not exist" Oct 31 01:21:43.895016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.894984 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 01:21:43.906991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:43.906964 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-mdw5x"] Oct 31 01:21:45.688924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:45.688880 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" path="/var/lib/kubelet/pods/d0c1d831-fefc-4d9e-9fc9-107235d04648/volumes" Oct 31 01:21:48.993443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:48.993402 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerName="main" containerID="cri-o://44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4" gracePeriod=45 Oct 31 01:21:49.153555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.153513 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerName="main" probeResult="failure" output="Get \"https://10.132.1.18:8000/ready\": dial tcp 10.132.1.18:8000: connect: connection refused" Oct 31 01:21:49.273277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.273251 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:21:49.381562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381529 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.381751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381588 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.381751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381621 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.381751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381674 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\" (UniqueName: \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.381751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381717 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.382028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381865 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\") pod \"f3859949-cb4a-4c68-acba-659c0eea5f16\" (UID: \"f3859949-cb4a-4c68-acba-659c0eea5f16\") " Oct 31 01:21:49.382028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:49.382028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.381976 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:49.382197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.382113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:49.382371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.382347 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.382371 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.382372 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.382532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.382382 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.384070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.384046 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:49.384070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.384056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:49.384195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.384103 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg" pod "f3859949-cb4a-4c68-acba-659c0eea5f16" (UID: "f3859949-cb4a-4c68-acba-659c0eea5f16"). InnerVolumeSpecName "kube-api-access-m4xfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:49.483378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.483343 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3859949-cb4a-4c68-acba-659c0eea5f16-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.483378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.483369 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3859949-cb4a-4c68-acba-659c0eea5f16-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.483378 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.483381 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xfg\" (UniqueName: \"kubernetes.io/projected/f3859949-cb4a-4c68-acba-659c0eea5f16-kube-api-access-m4xfg\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:49.881251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.879334 5011 generic.go:356] "Generic (PLEG): container finished" podID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerID="44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4" exitCode=0 Oct 31 01:21:49.881251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.879462 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" event={"ID":"f3859949-cb4a-4c68-acba-659c0eea5f16","Type":"ContainerDied","Data":"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4"} Oct 31 01:21:49.881251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.879478 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" Oct 31 01:21:49.881251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.879508 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn" event={"ID":"f3859949-cb4a-4c68-acba-659c0eea5f16","Type":"ContainerDied","Data":"bf49e93079387076444bbf074fffc603d735a130548a3e9de632f4b747ae89c3"} Oct 31 01:21:49.881251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.879538 5011 scope.go:122] "RemoveContainer" containerID="44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4" Oct 31 01:21:49.908243 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.908216 5011 scope.go:122] "RemoveContainer" containerID="44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4" Oct 31 01:21:49.908650 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:21:49.908627 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4\": container with ID starting with 44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4 not found: ID does not exist" containerID="44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4" Oct 31 01:21:49.908706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.908662 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4"} err="failed to get container status \"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4\": rpc error: code = NotFound desc = could not find container \"44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4\": container with ID starting with 44e3110fa2321e46e78c85ce2e30cb2413f362107b60a1851588d14b9023f6e4 not found: ID does not exist" Oct 31 01:21:49.939855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.939820 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:21:49.946105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:49.946045 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d10721/rate-test-model-d10721-kserve-65bc769fb5-gkkbn"] Oct 31 01:21:50.905938 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:50.905789 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:21:51.683798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:51.683764 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" path="/var/lib/kubelet/pods/f3859949-cb4a-4c68-acba-659c0eea5f16/volumes" Oct 31 01:21:52.058335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.058240 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht"] Oct 31 01:21:52.060133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060101 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerName="main" Oct 31 01:21:52.060363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060345 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerName="main" Oct 31 01:21:52.060502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060489 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" Oct 31 01:21:52.060603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060590 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" Oct 31 01:21:52.060779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060763 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" Oct 31 01:21:52.060895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.060882 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" Oct 31 01:21:52.061015 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061002 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5137c3f0-e155-4d4c-bf53-5cf18984b3dc" containerName="revoke-keys" Oct 31 01:21:52.061108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061095 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5137c3f0-e155-4d4c-bf53-5cf18984b3dc" containerName="revoke-keys" Oct 31 01:21:52.061241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061227 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerName="main" Oct 31 01:21:52.061350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061336 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerName="main" Oct 31 01:21:52.061711 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061688 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5137c3f0-e155-4d4c-bf53-5cf18984b3dc" containerName="revoke-keys" Oct 31 01:21:52.061857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061844 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72c884c-40e1-45d5-bb7a-6cf677d95b8e" containerName="main" Oct 31 01:21:52.061972 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.061959 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3859949-cb4a-4c68-acba-659c0eea5f16" containerName="main" Oct 31 01:21:52.062081 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.062067 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c1d831-fefc-4d9e-9fc9-107235d04648" containerName="istio-proxy" Oct 31 01:21:52.062222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.062192 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39ac115-645a-4611-8d44-d4aea5b4b13c" containerName="payload-processing" Oct 31 01:21:52.069548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.069525 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.086687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.086659 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht"] Oct 31 01:21:52.090900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.090876 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:52.097803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.097073 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:21:52.234114 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.234076 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.234355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.234133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.234355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.234307 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5cr\" (UniqueName: \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.335867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.335833 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.336023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.335878 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.336023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.335936 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5cr\" (UniqueName: \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.336659 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.336637 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.338416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.338389 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.358876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.358852 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5cr\" (UniqueName: \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.393663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.393624 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:52.704260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.704226 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht"] Oct 31 01:21:52.705801 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:21:52.705759 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6782c4e_4070_4b90_9256_60b489510f25.slice/crio-e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500 WatchSource:0}: Error finding container e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500: Status 404 returned error can't find the container with id e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500 Oct 31 01:21:52.942961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:52.942928 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" event={"ID":"c6782c4e-4070-4b90-9256-60b489510f25","Type":"ContainerStarted","Data":"e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500"} Oct 31 01:21:53.952517 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:53.952435 5011 generic.go:356] "Generic (PLEG): container finished" podID="c6782c4e-4070-4b90-9256-60b489510f25" containerID="b40365de0f93e74b63da8ccb8542d178c73e4658a56deb0f9755f825e9c11270" exitCode=0 Oct 31 01:21:53.952888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:53.952528 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" event={"ID":"c6782c4e-4070-4b90-9256-60b489510f25","Type":"ContainerDied","Data":"b40365de0f93e74b63da8ccb8542d178c73e4658a56deb0f9755f825e9c11270"} Oct 31 01:21:55.120394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.120365 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:55.155434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.155408 5011 scope.go:122] "RemoveContainer" containerID="3bb9d755549bc0d922ecedcf604e1882587fb24be7314cfe4549bd24cf7a1506" Oct 31 01:21:55.168448 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.168421 5011 scope.go:122] "RemoveContainer" containerID="ee1b42c2c3681f264a9463b9f1467ce808804843d1c0757f32d5adae9108a653" Oct 31 01:21:55.182598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.182577 5011 scope.go:122] "RemoveContainer" containerID="b3e3a3c1a9c3da396961dae043b7bb74ce49f9adf97d6a7f88786f4cadf2a8f8" Oct 31 01:21:55.196939 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.196914 5011 scope.go:122] "RemoveContainer" containerID="15dea1ac40a104254acfa5578eca1a1684895d6d8266d7d5a88adc9b58226d54" Oct 31 01:21:55.268515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.268483 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\") pod \"c6782c4e-4070-4b90-9256-60b489510f25\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " Oct 31 01:21:55.268728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.268706 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\") pod \"c6782c4e-4070-4b90-9256-60b489510f25\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " Oct 31 01:21:55.268809 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.268760 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\" (UniqueName: \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\") pod \"c6782c4e-4070-4b90-9256-60b489510f25\" (UID: \"c6782c4e-4070-4b90-9256-60b489510f25\") " Oct 31 01:21:55.269590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.269556 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca" pod "c6782c4e-4070-4b90-9256-60b489510f25" (UID: "c6782c4e-4070-4b90-9256-60b489510f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:55.271274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.271243 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca" pod "c6782c4e-4070-4b90-9256-60b489510f25" (UID: "c6782c4e-4070-4b90-9256-60b489510f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:55.271678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.271651 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr" pod "c6782c4e-4070-4b90-9256-60b489510f25" (UID: "c6782c4e-4070-4b90-9256-60b489510f25"). InnerVolumeSpecName "kube-api-access-2r5cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:55.370794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.370759 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:55.370794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.370789 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6782c4e-4070-4b90-9256-60b489510f25-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:55.370794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.370799 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r5cr\" (UniqueName: \"kubernetes.io/projected/c6782c4e-4070-4b90-9256-60b489510f25-kube-api-access-2r5cr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:21:55.976847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.976764 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" Oct 31 01:21:55.976847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.976796 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht" event={"ID":"c6782c4e-4070-4b90-9256-60b489510f25","Type":"ContainerDied","Data":"e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500"} Oct 31 01:21:55.977033 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:21:55.976871 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f90e66944250c081cb35d89b99506b6296311ff5899922a4a476ec29537500" Oct 31 01:22:00.595236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:00.595190 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb"] Oct 31 01:22:00.606940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:00.606912 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-714f8f-pqvwb"] Oct 31 01:22:01.693123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:01.693086 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5137c3f0-e155-4d4c-bf53-5cf18984b3dc" path="/var/lib/kubelet/pods/5137c3f0-e155-4d4c-bf53-5cf18984b3dc/volumes" Oct 31 01:22:01.948480 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:01.948395 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:22:02.356977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.356939 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:22:02.357481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.357466 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c6782c4e-4070-4b90-9256-60b489510f25" containerName="revoke-keys" Oct 31 01:22:02.357538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.357482 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6782c4e-4070-4b90-9256-60b489510f25" containerName="revoke-keys" Oct 31 01:22:02.357597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.357588 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6782c4e-4070-4b90-9256-60b489510f25" containerName="revoke-keys" Oct 31 01:22:02.361083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.361058 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.382172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.382138 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:22:02.453771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.453731 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454035 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454223 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrrf\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454563 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454546 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454687 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.454951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.454927 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.455127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.455108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.455278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.455261 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.471007 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.470965 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8f0029-openshift-default-dockercfg-g7gfd\"" Oct 31 01:22:02.556226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556172 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556270 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrrf\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556340 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556390 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556415 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556456 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556530 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556569 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556726 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.556859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.556751 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.557096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.557067 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.557186 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.557170 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.557311 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.557286 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.559012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.558993 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.559687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.559670 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.594888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.594854 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.598922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.598893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrrf\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:02.676054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:02.675989 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:03.030991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:03.030947 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:22:03.032954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:03.032910 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:03.033056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:03.033004 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:03.033056 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:03.033047 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:04.066610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.066565 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" event={"ID":"d31bbad8-d7c6-4092-a7ee-e7484076a9a3","Type":"ContainerStarted","Data":"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d"} Oct 31 01:22:04.067087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.066625 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" event={"ID":"d31bbad8-d7c6-4092-a7ee-e7484076a9a3","Type":"ContainerStarted","Data":"dac1dffa7b00c53462b6dea69aaeb27313121314449629e8b1b68160b35e4e8b"} Oct 31 01:22:04.190231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.169341 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podStartSLOduration=2.169318128 podStartE2EDuration="2.169318128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:04.130585289 +0000 UTC m=+3792.579672929" watchObservedRunningTime="2026-10-31 01:22:04.169318128 +0000 UTC m=+3792.618405769" Oct 31 01:22:04.676289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.676245 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:04.677628 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.677598 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 01:22:04.677786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:04.677655 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 01:22:05.676568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:05.676532 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 01:22:05.676978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:05.676585 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 01:22:06.676478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:06.676432 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 01:22:06.676668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:06.676492 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 01:22:07.677431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:07.677391 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 01:22:07.677947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:07.677453 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 01:22:08.676408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:08.676372 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 01:22:08.676581 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:08.676430 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 01:22:09.090312 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.090273 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj"] Oct 31 01:22:09.094754 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.094722 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.115877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.115849 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj"] Oct 31 01:22:09.125729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.125676 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:09.125917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.125683 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:22:09.177424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.177384 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.177609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.177499 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.177660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.177622 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9726h\" (UniqueName: \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.278974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.278932 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.279156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.279017 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.279265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.279174 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9726h\" (UniqueName: \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.279745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.279715 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.282139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.282100 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.303139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.303104 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9726h\" (UniqueName: \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\") pod \"maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.414053 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.413966 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:09.683383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.683302 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:09.683779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.683756 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:09.684680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.684653 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:22:09.737497 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:09.737464 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4399fc62_7861_44de_9576_9e71ce1d252d.slice/crio-b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700 WatchSource:0}: Error finding container b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700: Status 404 returned error can't find the container with id b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700 Oct 31 01:22:09.746804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:09.745808 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj"] Oct 31 01:22:10.116275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:10.116241 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" event={"ID":"4399fc62-7861-44de-9576-9e71ce1d252d","Type":"ContainerStarted","Data":"b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700"} Oct 31 01:22:11.126428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.126391 5011 generic.go:356] "Generic (PLEG): container finished" podID="4399fc62-7861-44de-9576-9e71ce1d252d" containerID="195510f388609b9d63f1a53f439305e2a5c15ee675349e8eef69ad7a0697a75f" exitCode=0 Oct 31 01:22:11.126867 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.126480 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" event={"ID":"4399fc62-7861-44de-9576-9e71ce1d252d","Type":"ContainerDied","Data":"195510f388609b9d63f1a53f439305e2a5c15ee675349e8eef69ad7a0697a75f"} Oct 31 01:22:11.226954 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.226819 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:22:11.227141 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.227076 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" containerID="cri-o://ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c" gracePeriod=30 Oct 31 01:22:11.806137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.806092 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg"] Oct 31 01:22:11.810428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.810403 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:11.829355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.829323 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg"] Oct 31 01:22:11.910759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.910727 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:11.910926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.910770 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:11.910926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:11.910897 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9cc\" (UniqueName: \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.012748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.012706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.012926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.012768 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.012926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.012839 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9cc\" (UniqueName: \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.013500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.013471 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.015139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.015119 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.037453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.037423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9cc\" (UniqueName: \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.124701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.124676 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:12.369820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.369793 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:12.429768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.429730 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\" (UniqueName: \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\") pod \"4399fc62-7861-44de-9576-9e71ce1d252d\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " Oct 31 01:22:12.429960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.429833 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\") pod \"4399fc62-7861-44de-9576-9e71ce1d252d\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " Oct 31 01:22:12.429960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.429878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\") pod \"4399fc62-7861-44de-9576-9e71ce1d252d\" (UID: \"4399fc62-7861-44de-9576-9e71ce1d252d\") " Oct 31 01:22:12.430478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.430310 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca" pod "4399fc62-7861-44de-9576-9e71ce1d252d" (UID: "4399fc62-7861-44de-9576-9e71ce1d252d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:12.432348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.432321 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca" pod "4399fc62-7861-44de-9576-9e71ce1d252d" (UID: "4399fc62-7861-44de-9576-9e71ce1d252d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:12.432673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.432649 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h" pod "4399fc62-7861-44de-9576-9e71ce1d252d" (UID: "4399fc62-7861-44de-9576-9e71ce1d252d"). InnerVolumeSpecName "kube-api-access-9726h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:12.444372 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:12.443731 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e14973c_d6d7_4605_8e02_0523844b8a5f.slice/crio-1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7 WatchSource:0}: Error finding container 1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7: Status 404 returned error can't find the container with id 1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7 Oct 31 01:22:12.444372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.443857 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg"] Oct 31 01:22:12.531462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.531429 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9726h\" (UniqueName: \"kubernetes.io/projected/4399fc62-7861-44de-9576-9e71ce1d252d-kube-api-access-9726h\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:12.531462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.531455 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:12.531462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:12.531466 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4399fc62-7861-44de-9576-9e71ce1d252d-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:13.153228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.152414 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" Oct 31 01:22:13.153228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.152425 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj" event={"ID":"4399fc62-7861-44de-9576-9e71ce1d252d","Type":"ContainerDied","Data":"b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700"} Oct 31 01:22:13.153228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.152484 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19609690b93d3d7c8b8440a372554dd1e6743851f9b425c7a965744ef8ea700" Oct 31 01:22:13.154582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.154546 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" event={"ID":"0e14973c-d6d7-4605-8e02-0523844b8a5f","Type":"ContainerStarted","Data":"03bed4c097c154434620f1e53bdb22680b3c0a0d3150060fbb91abe5c37d2226"} Oct 31 01:22:13.154793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.154600 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" event={"ID":"0e14973c-d6d7-4605-8e02-0523844b8a5f","Type":"ContainerStarted","Data":"1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7"} Oct 31 01:22:13.197288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.197235 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" podStartSLOduration=2.197219262 podStartE2EDuration="2.197219262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:13.195529083 +0000 UTC m=+3801.644616722" watchObservedRunningTime="2026-10-31 01:22:13.197219262 +0000 UTC m=+3801.646306891" Oct 31 01:22:13.788370 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.788276 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 01:22:13.792858 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.792829 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 01:22:13.809679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:13.809652 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 01:22:14.164444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:14.164406 5011 generic.go:356] "Generic (PLEG): container finished" podID="0e14973c-d6d7-4605-8e02-0523844b8a5f" containerID="03bed4c097c154434620f1e53bdb22680b3c0a0d3150060fbb91abe5c37d2226" exitCode=0 Oct 31 01:22:14.164819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:14.164507 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" event={"ID":"0e14973c-d6d7-4605-8e02-0523844b8a5f","Type":"ContainerDied","Data":"03bed4c097c154434620f1e53bdb22680b3c0a0d3150060fbb91abe5c37d2226"} Oct 31 01:22:14.852845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:14.852808 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:22:14.856235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:14.852872 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:22:15.337301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.337276 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:15.363163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.363138 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\") pod \"0e14973c-d6d7-4605-8e02-0523844b8a5f\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " Oct 31 01:22:15.363320 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.363182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\") pod \"0e14973c-d6d7-4605-8e02-0523844b8a5f\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " Oct 31 01:22:15.363372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.363328 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\" (UniqueName: \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\") pod \"0e14973c-d6d7-4605-8e02-0523844b8a5f\" (UID: \"0e14973c-d6d7-4605-8e02-0523844b8a5f\") " Oct 31 01:22:15.363550 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.363524 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca" pod "0e14973c-d6d7-4605-8e02-0523844b8a5f" (UID: "0e14973c-d6d7-4605-8e02-0523844b8a5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:15.363852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.363829 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:15.365374 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.365350 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca" pod "0e14973c-d6d7-4605-8e02-0523844b8a5f" (UID: "0e14973c-d6d7-4605-8e02-0523844b8a5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:15.365459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.365439 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc" pod "0e14973c-d6d7-4605-8e02-0523844b8a5f" (UID: "0e14973c-d6d7-4605-8e02-0523844b8a5f"). InnerVolumeSpecName "kube-api-access-vh9cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:15.465816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.465717 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14973c-d6d7-4605-8e02-0523844b8a5f-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:15.465816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:15.465756 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh9cc\" (UniqueName: \"kubernetes.io/projected/0e14973c-d6d7-4605-8e02-0523844b8a5f-kube-api-access-vh9cc\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.185353 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.185321 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" Oct 31 01:22:16.185863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.185350 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg" event={"ID":"0e14973c-d6d7-4605-8e02-0523844b8a5f","Type":"ContainerDied","Data":"1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7"} Oct 31 01:22:16.185863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.185402 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f487167b5b9542d592a568911cd4ffeef6d28bcae824409b0fd5cb896ef9ee7" Oct 31 01:22:16.506073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.506047 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:22:16.578251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578195 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578280 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578359 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578388 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578438 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578489 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578535 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578630 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\") pod \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\" (UID: \"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1\") " Oct 31 01:22:16.578895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.578673 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.579240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.579195 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.579455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.579434 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.579530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.579447 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.579645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.579605 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:16.579853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.579831 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.581290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.581006 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:16.581290 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.581055 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "kube-api-access-89r89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:16.582796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.582766 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.582909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.582846 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token" pod "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" (UID: "d70aafa2-aa36-4ca0-b27e-a1fcb68288e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:16.680763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680728 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680767 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680777 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680786 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680796 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680805 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680813 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89r89\" (UniqueName: \"kubernetes.io/projected/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-kube-api-access-89r89\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.680964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:16.680821 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:17.195333 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.195297 5011 generic.go:356] "Generic (PLEG): container finished" podID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerID="ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c" exitCode=0 Oct 31 01:22:17.195749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.195342 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" Oct 31 01:22:17.195749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.195380 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerDied","Data":"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c"} Oct 31 01:22:17.195749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.195437 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn" event={"ID":"d70aafa2-aa36-4ca0-b27e-a1fcb68288e1","Type":"ContainerDied","Data":"f3c771188eacf44e2109fd792ff6b722bca63025ec1fc104bf5e34c752685b8d"} Oct 31 01:22:17.195749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.195465 5011 scope.go:122] "RemoveContainer" containerID="ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c" Oct 31 01:22:17.207047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.207026 5011 scope.go:122] "RemoveContainer" containerID="112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740" Oct 31 01:22:17.216616 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.216598 5011 scope.go:122] "RemoveContainer" containerID="ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c" Oct 31 01:22:17.216895 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:17.216876 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c\": container with ID starting with ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c not found: ID does not exist" containerID="ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c" Oct 31 01:22:17.216956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.216904 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c"} err="failed to get container status \"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c\": rpc error: code = NotFound desc = could not find container \"ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c\": container with ID starting with ada98c0c820fecd74d592ab723ed952d79a926f3e464ea9f1bd75624427de64c not found: ID does not exist" Oct 31 01:22:17.216956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.216924 5011 scope.go:122] "RemoveContainer" containerID="112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740" Oct 31 01:22:17.217192 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:17.217166 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740\": container with ID starting with 112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740 not found: ID does not exist" containerID="112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740" Oct 31 01:22:17.217251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.217195 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740"} err="failed to get container status \"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740\": rpc error: code = NotFound desc = could not find container \"112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740\": container with ID starting with 112c4907cf9714cf0a17bf67d1fde7fed51bef914244e2e1faf9c9ebb059c740 not found: ID does not exist" Oct 31 01:22:17.270657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.270631 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:22:17.282362 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.282337 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-h7ntn"] Oct 31 01:22:17.684502 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:17.684463 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" path="/var/lib/kubelet/pods/d70aafa2-aa36-4ca0-b27e-a1fcb68288e1/volumes" Oct 31 01:22:25.194181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:25.194144 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:22:27.943323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:27.943286 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:22:27.944048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:27.944003 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerName="maas-api" containerID="cri-o://0d09a58ec65132ffd49e38f78375cc9fbffa15f1ccd34ba76a26fb024bed0ad3" gracePeriod=30 Oct 31 01:22:28.399802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.399768 5011 generic.go:356] "Generic (PLEG): container finished" podID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerID="0d09a58ec65132ffd49e38f78375cc9fbffa15f1ccd34ba76a26fb024bed0ad3" exitCode=0 Oct 31 01:22:28.399949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.399811 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" event={"ID":"068a4a68-e133-4cc8-9a19-dcb0ad036218","Type":"ContainerDied","Data":"0d09a58ec65132ffd49e38f78375cc9fbffa15f1ccd34ba76a26fb024bed0ad3"} Oct 31 01:22:28.405380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.405352 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:22:28.468546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.468513 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\") pod \"068a4a68-e133-4cc8-9a19-dcb0ad036218\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " Oct 31 01:22:28.468720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.468568 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\" (UniqueName: \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\") pod \"068a4a68-e133-4cc8-9a19-dcb0ad036218\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " Oct 31 01:22:28.468720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.468683 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\") pod \"068a4a68-e133-4cc8-9a19-dcb0ad036218\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " Oct 31 01:22:28.468853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.468807 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\") pod \"068a4a68-e133-4cc8-9a19-dcb0ad036218\" (UID: \"068a4a68-e133-4cc8-9a19-dcb0ad036218\") " Oct 31 01:22:28.469413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.469377 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca" pod "068a4a68-e133-4cc8-9a19-dcb0ad036218" (UID: "068a4a68-e133-4cc8-9a19-dcb0ad036218"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:28.471807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.471766 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls" pod "068a4a68-e133-4cc8-9a19-dcb0ad036218" (UID: "068a4a68-e133-4cc8-9a19-dcb0ad036218"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:28.471807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.471800 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q" pod "068a4a68-e133-4cc8-9a19-dcb0ad036218" (UID: "068a4a68-e133-4cc8-9a19-dcb0ad036218"). InnerVolumeSpecName "kube-api-access-d7c5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:28.473476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.473444 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs" pod "068a4a68-e133-4cc8-9a19-dcb0ad036218" (UID: "068a4a68-e133-4cc8-9a19-dcb0ad036218"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:28.570351 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.570311 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/068a4a68-e133-4cc8-9a19-dcb0ad036218-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:28.570351 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.570353 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:28.570592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.570369 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/068a4a68-e133-4cc8-9a19-dcb0ad036218-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:28.570592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.570384 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7c5q\" (UniqueName: \"kubernetes.io/projected/068a4a68-e133-4cc8-9a19-dcb0ad036218-kube-api-access-d7c5q\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:28.656616 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.656517 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht"] Oct 31 01:22:28.662165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.662138 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d10721-4nhht"] Oct 31 01:22:28.809880 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.809840 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" containerID="cri-o://ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675" gracePeriod=45 Oct 31 01:22:28.813712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.813672 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" containerID="cri-o://56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c" gracePeriod=45 Oct 31 01:22:28.830818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:28.830781 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" containerID="cri-o://0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef" gracePeriod=45 Oct 31 01:22:28.881625 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:28.881600 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a96945f_f289_4cd4_ad20_d21439cb75e9.slice/crio-0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:22:29.158852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.158820 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 01:22:29.171107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.171026 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 01:22:29.196359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.196335 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 01:22:29.277149 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277116 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.277332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277156 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\" (UniqueName: \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277176 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.277332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277298 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\" (UniqueName: \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277332 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\" (UniqueName: \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277362 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277401 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277505 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277551 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.277590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277590 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277633 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277705 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277756 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277803 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277837 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277869 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\") pod \"8a96945f-f289-4cd4-ad20-d21439cb75e9\" (UID: \"8a96945f-f289-4cd4-ad20-d21439cb75e9\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277896 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\") pod \"9eee9175-5c96-48c2-847d-ffb24b312ba3\" (UID: \"9eee9175-5c96-48c2-847d-ffb24b312ba3\") " Oct 31 01:22:29.277948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.277934 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\") pod \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\" (UID: \"4cefe0ca-326c-4a75-b158-60ce35efaa4f\") " Oct 31 01:22:29.278429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278043 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.278429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278054 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.278429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278354 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.278608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278545 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.278678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278644 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.278678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278660 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.278678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278660 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.278678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278673 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.278987 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.278959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.279175 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.279151 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.279452 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.279412 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.279638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.279616 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284359 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284374 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284399 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "kube-api-access-4lvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284445 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284457 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "kube-api-access-mld52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284473 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "kube-api-access-5mcj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284498 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm" pod "4cefe0ca-326c-4a75-b158-60ce35efaa4f" (UID: "4cefe0ca-326c-4a75-b158-60ce35efaa4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284514 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm" pod "9eee9175-5c96-48c2-847d-ffb24b312ba3" (UID: "9eee9175-5c96-48c2-847d-ffb24b312ba3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.284701 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.284524 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm" pod "8a96945f-f289-4cd4-ad20-d21439cb75e9" (UID: "8a96945f-f289-4cd4-ad20-d21439cb75e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.379962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379926 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eee9175-5c96-48c2-847d-ffb24b312ba3-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.379962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379959 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.379962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379968 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379976 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379984 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.379992 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a96945f-f289-4cd4-ad20-d21439cb75e9-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380000 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380007 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cefe0ca-326c-4a75-b158-60ce35efaa4f-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380015 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cefe0ca-326c-4a75-b158-60ce35efaa4f-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380024 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mld52\" (UniqueName: \"kubernetes.io/projected/8a96945f-f289-4cd4-ad20-d21439cb75e9-kube-api-access-mld52\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380034 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eee9175-5c96-48c2-847d-ffb24b312ba3-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380042 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcj5\" (UniqueName: \"kubernetes.io/projected/4cefe0ca-326c-4a75-b158-60ce35efaa4f-kube-api-access-5mcj5\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380050 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvh7\" (UniqueName: \"kubernetes.io/projected/9eee9175-5c96-48c2-847d-ffb24b312ba3-kube-api-access-4lvh7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.380216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.380058 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a96945f-f289-4cd4-ad20-d21439cb75e9-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.411030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.410996 5011 generic.go:356] "Generic (PLEG): container finished" podID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerID="56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c" exitCode=0 Oct 31 01:22:29.411248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.411045 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" Oct 31 01:22:29.411248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.411077 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" event={"ID":"9eee9175-5c96-48c2-847d-ffb24b312ba3","Type":"ContainerDied","Data":"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c"} Oct 31 01:22:29.411248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.411127 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" event={"ID":"9eee9175-5c96-48c2-847d-ffb24b312ba3","Type":"ContainerDied","Data":"eae79885cc919c63f50e2eb7640c9395679b2e98547958f030fbf592e007bdf0"} Oct 31 01:22:29.411248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.411151 5011 scope.go:122] "RemoveContainer" containerID="56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c" Oct 31 01:22:29.412923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.412881 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" event={"ID":"068a4a68-e133-4cc8-9a19-dcb0ad036218","Type":"ContainerDied","Data":"ca88484a291489729048eeea2919394a24804d48905574f160c2840eb30e32e5"} Oct 31 01:22:29.413074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.412890 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" Oct 31 01:22:29.414692 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.414668 5011 generic.go:356] "Generic (PLEG): container finished" podID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerID="ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675" exitCode=0 Oct 31 01:22:29.414803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.414719 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" Oct 31 01:22:29.414803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.414746 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" event={"ID":"4cefe0ca-326c-4a75-b158-60ce35efaa4f","Type":"ContainerDied","Data":"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675"} Oct 31 01:22:29.414803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.414786 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" event={"ID":"4cefe0ca-326c-4a75-b158-60ce35efaa4f","Type":"ContainerDied","Data":"0e709e99121fd7dec4855857e66ed79ade9a7d0404d297e3f4bbeaa2bf8026ae"} Oct 31 01:22:29.416564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.416541 5011 generic.go:356] "Generic (PLEG): container finished" podID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerID="0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef" exitCode=0 Oct 31 01:22:29.416672 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.416582 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" Oct 31 01:22:29.416672 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.416608 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" event={"ID":"8a96945f-f289-4cd4-ad20-d21439cb75e9","Type":"ContainerDied","Data":"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef"} Oct 31 01:22:29.416672 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.416635 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" event={"ID":"8a96945f-f289-4cd4-ad20-d21439cb75e9","Type":"ContainerDied","Data":"9a501c87adea97932ad64d048dce7d2e002025c8bf85a217289f1ce57df31b90"} Oct 31 01:22:29.423419 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.423398 5011 scope.go:122] "RemoveContainer" containerID="56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c" Oct 31 01:22:29.423789 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:29.423724 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c\": container with ID starting with 56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c not found: ID does not exist" containerID="56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c" Oct 31 01:22:29.423789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.423765 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c"} err="failed to get container status \"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c\": rpc error: code = NotFound desc = could not find container \"56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c\": container with ID starting with 56d581574b20991332c9034e0787e5e452227170f9579a3af54f5bc4286b597c not found: ID does not exist" Oct 31 01:22:29.423945 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.423788 5011 scope.go:122] "RemoveContainer" containerID="0d09a58ec65132ffd49e38f78375cc9fbffa15f1ccd34ba76a26fb024bed0ad3" Oct 31 01:22:29.437331 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.437307 5011 scope.go:122] "RemoveContainer" containerID="ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675" Oct 31 01:22:29.449577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.449495 5011 scope.go:122] "RemoveContainer" containerID="ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675" Oct 31 01:22:29.449901 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:29.449875 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675\": container with ID starting with ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675 not found: ID does not exist" containerID="ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675" Oct 31 01:22:29.450011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.449915 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675"} err="failed to get container status \"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675\": rpc error: code = NotFound desc = could not find container \"ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675\": container with ID starting with ceebc4cd02475afe1083019c6c3bc7d5dbdbfff8f1cf28769115cad834313675 not found: ID does not exist" Oct 31 01:22:29.450011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.449937 5011 scope.go:122] "RemoveContainer" containerID="0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef" Oct 31 01:22:29.464039 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.464016 5011 scope.go:122] "RemoveContainer" containerID="0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef" Oct 31 01:22:29.464422 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:29.464395 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef\": container with ID starting with 0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef not found: ID does not exist" containerID="0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef" Oct 31 01:22:29.464504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.464430 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef"} err="failed to get container status \"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef\": rpc error: code = NotFound desc = could not find container \"0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef\": container with ID starting with 0eca27d8cf7d0a141839d781c8bd776499fcc8128ebd2f73db6372058e8ca1ef not found: ID does not exist" Oct 31 01:22:29.500474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.500448 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 01:22:29.504823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.504799 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9"] Oct 31 01:22:29.555828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.555798 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:22:29.559823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.559801 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz"] Oct 31 01:22:29.594323 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.594302 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 01:22:29.598190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.598166 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx"] Oct 31 01:22:29.632327 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.632304 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 01:22:29.639155 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.639134 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl"] Oct 31 01:22:29.682663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.682602 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" path="/var/lib/kubelet/pods/068a4a68-e133-4cc8-9a19-dcb0ad036218/volumes" Oct 31 01:22:29.683110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.683094 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" path="/var/lib/kubelet/pods/4cefe0ca-326c-4a75-b158-60ce35efaa4f/volumes" Oct 31 01:22:29.683528 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.683507 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" path="/var/lib/kubelet/pods/8a96945f-f289-4cd4-ad20-d21439cb75e9/volumes" Oct 31 01:22:29.684173 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.684159 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" path="/var/lib/kubelet/pods/9eee9175-5c96-48c2-847d-ffb24b312ba3/volumes" Oct 31 01:22:29.684600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:29.684585 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6782c4e-4070-4b90-9256-60b489510f25" path="/var/lib/kubelet/pods/c6782c4e-4070-4b90-9256-60b489510f25/volumes" Oct 31 01:22:30.093875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:30.093839 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/simulator-w6-be301b-kserve-74c7795fc8-zdshx" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 01:22:30.094079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:30.093850 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-2-w6-be301b-kserve-75dc7464d9-2cbnl" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:22:30.102804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:30.102777 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/distinct-w6-be301b-kserve-74dc58d6f8-fzfz9" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:22:31.205060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:31.205016 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d10721-d65db866d-jcsgz" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.221:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:22:31.947487 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:31.947453 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:22:35.891441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:35.891341 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:22:35.891916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:35.891579 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" containerID="cri-o://a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2" gracePeriod=30 Oct 31 01:22:36.509403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509371 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw"] Oct 31 01:22:36.509850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509836 5011 cpu_manager.go:414] "containerMap: removing container" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerName="maas-api" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509851 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerName="maas-api" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509871 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509878 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509887 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509892 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509903 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4399fc62-7861-44de-9576-9e71ce1d252d" containerName="revoke-keys" Oct 31 01:22:36.509909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509908 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4399fc62-7861-44de-9576-9e71ce1d252d" containerName="revoke-keys" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509922 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509926 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509936 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0e14973c-d6d7-4605-8e02-0523844b8a5f" containerName="revoke-keys" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509940 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e14973c-d6d7-4605-8e02-0523844b8a5f" containerName="revoke-keys" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509951 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509955 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509963 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.509968 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510050 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510058 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e14973c-d6d7-4605-8e02-0523844b8a5f" containerName="revoke-keys" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510065 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a96945f-f289-4cd4-ad20-d21439cb75e9" containerName="main" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510071 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70aafa2-aa36-4ca0-b27e-a1fcb68288e1" containerName="istio-proxy" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510078 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="068a4a68-e133-4cc8-9a19-dcb0ad036218" containerName="maas-api" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510086 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eee9175-5c96-48c2-847d-ffb24b312ba3" containerName="main" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510092 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cefe0ca-326c-4a75-b158-60ce35efaa4f" containerName="main" Oct 31 01:22:36.510122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.510098 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4399fc62-7861-44de-9576-9e71ce1d252d" containerName="revoke-keys" Oct 31 01:22:36.515103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.515082 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.535123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.535096 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw"] Oct 31 01:22:36.591137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.591111 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:36.591321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.591251 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:22:36.665030 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.664998 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.665191 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.665071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.665268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.665193 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6sv\" (UniqueName: \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.766420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.766335 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.766595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.766434 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.766595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.766454 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6sv\" (UniqueName: \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.767198 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.767169 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.768993 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.768967 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.791434 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.791400 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6sv\" (UniqueName: \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:36.837339 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:36.837305 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:37.144662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.144624 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw"] Oct 31 01:22:37.145087 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:37.144942 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3d33b6_2ecb_4633_95a7_9456e25682f7.slice/crio-ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853 WatchSource:0}: Error finding container ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853: Status 404 returned error can't find the container with id ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853 Oct 31 01:22:37.209105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.209074 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns"] Oct 31 01:22:37.214336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.214304 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.232274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.232250 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns"] Oct 31 01:22:37.252055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.252024 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr"] Oct 31 01:22:37.256952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.256929 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.259673 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.259647 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr"] Oct 31 01:22:37.271655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.271628 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.271772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.271754 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.271903 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.271870 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4g4d\" (UniqueName: \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.372614 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372578 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.372765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372689 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.372765 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372753 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.372894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372863 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.372955 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372941 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4g4d\" (UniqueName: \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.373012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.372978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdp9\" (UniqueName: \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.373566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.373537 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.375076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.375057 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.396645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.396584 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4g4d\" (UniqueName: \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\") pod \"maas-api-revoke-sub-523fc951-v4nns\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.474540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.474508 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.474655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.474598 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.474707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.474666 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdp9\" (UniqueName: \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.475198 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.475174 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.476721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.476699 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.487592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.487561 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" event={"ID":"6a3d33b6-2ecb-4633-95a7-9456e25682f7","Type":"ContainerStarted","Data":"dea4157af9b7042bb9ef0aa9e144ba05a6fad53f3c1d87bf31de6cb87b6a8758"} Oct 31 01:22:37.487705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.487618 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" event={"ID":"6a3d33b6-2ecb-4633-95a7-9456e25682f7","Type":"ContainerStarted","Data":"ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853"} Oct 31 01:22:37.498200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.498176 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdp9\" (UniqueName: \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\") pod \"maas-api-revoke-sub-448d4daa-hsfcr\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.529843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.529819 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:37.530467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.530430 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" podStartSLOduration=1.530419591 podStartE2EDuration="1.530419591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:37.528362973 +0000 UTC m=+3825.977450612" watchObservedRunningTime="2026-10-31 01:22:37.530419591 +0000 UTC m=+3825.979507271" Oct 31 01:22:37.573559 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.573383 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:37.841595 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:37.841565 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e45ad7_1498_4d79_96ab_139941c2dfa9.slice/crio-79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45 WatchSource:0}: Error finding container 79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45: Status 404 returned error can't find the container with id 79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45 Oct 31 01:22:37.841989 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.841966 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns"] Oct 31 01:22:37.914397 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:37.914357 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee054917_39b6_4e9f_bbce_f3ef0f2a5f0a.slice/crio-5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560 WatchSource:0}: Error finding container 5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560: Status 404 returned error can't find the container with id 5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560 Oct 31 01:22:37.914976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:37.914888 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr"] Oct 31 01:22:38.496845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.496806 5011 generic.go:356] "Generic (PLEG): container finished" podID="6a3d33b6-2ecb-4633-95a7-9456e25682f7" containerID="dea4157af9b7042bb9ef0aa9e144ba05a6fad53f3c1d87bf31de6cb87b6a8758" exitCode=0 Oct 31 01:22:38.497281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.496886 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" event={"ID":"6a3d33b6-2ecb-4633-95a7-9456e25682f7","Type":"ContainerDied","Data":"dea4157af9b7042bb9ef0aa9e144ba05a6fad53f3c1d87bf31de6cb87b6a8758"} Oct 31 01:22:38.498592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.498557 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" event={"ID":"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a","Type":"ContainerStarted","Data":"fd931abb9120fa1600b72c5b765f983f085f8a82a9afe306556566f53e595152"} Oct 31 01:22:38.498725 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.498601 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" event={"ID":"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a","Type":"ContainerStarted","Data":"5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560"} Oct 31 01:22:38.500269 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.500182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" event={"ID":"04e45ad7-1498-4d79-96ab-139941c2dfa9","Type":"ContainerStarted","Data":"c5f72c73a79e9a0e154e6e86bfe75d5e8304771de399ad8de5bd1c0214aedf2b"} Oct 31 01:22:38.500269 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.500252 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" event={"ID":"04e45ad7-1498-4d79-96ab-139941c2dfa9","Type":"ContainerStarted","Data":"79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45"} Oct 31 01:22:38.585295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.585237 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" podStartSLOduration=1.585201463 podStartE2EDuration="1.585201463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:38.580849595 +0000 UTC m=+3827.029937235" watchObservedRunningTime="2026-10-31 01:22:38.585201463 +0000 UTC m=+3827.034289102" Oct 31 01:22:38.625779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:38.625728 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" podStartSLOduration=1.6257134610000001 podStartE2EDuration="1.625713461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:38.622925077 +0000 UTC m=+3827.072012716" watchObservedRunningTime="2026-10-31 01:22:38.625713461 +0000 UTC m=+3827.074801138" Oct 31 01:22:39.508757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.508717 5011 generic.go:356] "Generic (PLEG): container finished" podID="ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" containerID="fd931abb9120fa1600b72c5b765f983f085f8a82a9afe306556566f53e595152" exitCode=0 Oct 31 01:22:39.509176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.508794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" event={"ID":"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a","Type":"ContainerDied","Data":"fd931abb9120fa1600b72c5b765f983f085f8a82a9afe306556566f53e595152"} Oct 31 01:22:39.510930 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.510899 5011 generic.go:356] "Generic (PLEG): container finished" podID="04e45ad7-1498-4d79-96ab-139941c2dfa9" containerID="c5f72c73a79e9a0e154e6e86bfe75d5e8304771de399ad8de5bd1c0214aedf2b" exitCode=0 Oct 31 01:22:39.511082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.510978 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" event={"ID":"04e45ad7-1498-4d79-96ab-139941c2dfa9","Type":"ContainerDied","Data":"c5f72c73a79e9a0e154e6e86bfe75d5e8304771de399ad8de5bd1c0214aedf2b"} Oct 31 01:22:39.697097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.697070 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:39.812488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.812403 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\") pod \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " Oct 31 01:22:39.812488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.812483 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\" (UniqueName: \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\") pod \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " Oct 31 01:22:39.812679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.812541 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\") pod \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\" (UID: \"6a3d33b6-2ecb-4633-95a7-9456e25682f7\") " Oct 31 01:22:39.812968 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.812941 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca" pod "6a3d33b6-2ecb-4633-95a7-9456e25682f7" (UID: "6a3d33b6-2ecb-4633-95a7-9456e25682f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:39.814582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.814544 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca" pod "6a3d33b6-2ecb-4633-95a7-9456e25682f7" (UID: "6a3d33b6-2ecb-4633-95a7-9456e25682f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:39.814709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.814690 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv" pod "6a3d33b6-2ecb-4633-95a7-9456e25682f7" (UID: "6a3d33b6-2ecb-4633-95a7-9456e25682f7"). InnerVolumeSpecName "kube-api-access-bm6sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:39.913592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.913560 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:39.913592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.913592 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d33b6-2ecb-4633-95a7-9456e25682f7-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:39.913781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:39.913608 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6sv\" (UniqueName: \"kubernetes.io/projected/6a3d33b6-2ecb-4633-95a7-9456e25682f7-kube-api-access-bm6sv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.007066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.007029 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth"] Oct 31 01:22:40.007632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.007610 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6a3d33b6-2ecb-4633-95a7-9456e25682f7" containerName="revoke-keys" Oct 31 01:22:40.007632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.007631 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3d33b6-2ecb-4633-95a7-9456e25682f7" containerName="revoke-keys" Oct 31 01:22:40.007843 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.007828 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3d33b6-2ecb-4633-95a7-9456e25682f7" containerName="revoke-keys" Oct 31 01:22:40.011469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.011449 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.035977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.035374 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth"] Oct 31 01:22:40.115555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.115526 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.115703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.115655 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.115703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.115690 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqn7t\" (UniqueName: \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.216715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.216682 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.216879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.216724 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqn7t\" (UniqueName: \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.216879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.216843 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.217474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.217444 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.219441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.219412 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.240885 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.240854 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqn7t\" (UniqueName: \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\") pod \"maas-api-revoke-keys-e2e-ipp-a-114c95-59wth\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.325170 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.325145 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:40.519600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.519424 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" Oct 31 01:22:40.519600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.519457 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw" event={"ID":"6a3d33b6-2ecb-4633-95a7-9456e25682f7","Type":"ContainerDied","Data":"ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853"} Oct 31 01:22:40.519600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.519528 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2da1c7e25008fbb94a936f16c75e6b834d6ea331da3a15f0ba25c285bd2853" Oct 31 01:22:40.650784 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.650749 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth"] Oct 31 01:22:40.682365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.682338 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:40.708694 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.708671 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:40.823538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823490 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\") pod \"04e45ad7-1498-4d79-96ab-139941c2dfa9\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " Oct 31 01:22:40.823724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823598 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\") pod \"04e45ad7-1498-4d79-96ab-139941c2dfa9\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " Oct 31 01:22:40.823724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823622 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\") pod \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " Oct 31 01:22:40.823724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823652 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\") pod \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " Oct 31 01:22:40.823920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\" (UniqueName: \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\") pod \"04e45ad7-1498-4d79-96ab-139941c2dfa9\" (UID: \"04e45ad7-1498-4d79-96ab-139941c2dfa9\") " Oct 31 01:22:40.823920 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.823834 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\" (UniqueName: \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\") pod \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\" (UID: \"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a\") " Oct 31 01:22:40.824394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.824324 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca" pod "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" (UID: "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:40.824631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.824604 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca" pod "04e45ad7-1498-4d79-96ab-139941c2dfa9" (UID: "04e45ad7-1498-4d79-96ab-139941c2dfa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:40.826259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.826201 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d" pod "04e45ad7-1498-4d79-96ab-139941c2dfa9" (UID: "04e45ad7-1498-4d79-96ab-139941c2dfa9"). InnerVolumeSpecName "kube-api-access-h4g4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:40.826431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.826374 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca" pod "04e45ad7-1498-4d79-96ab-139941c2dfa9" (UID: "04e45ad7-1498-4d79-96ab-139941c2dfa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:40.826544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.826528 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca" pod "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" (UID: "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:40.826848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.826817 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9" pod "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" (UID: "ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a"). InnerVolumeSpecName "kube-api-access-mfdp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:40.925608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925580 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.925608 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925611 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e45ad7-1498-4d79-96ab-139941c2dfa9-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.925736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925622 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.925736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925631 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.925736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925640 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4g4d\" (UniqueName: \"kubernetes.io/projected/04e45ad7-1498-4d79-96ab-139941c2dfa9-kube-api-access-h4g4d\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:40.925736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:40.925648 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdp9\" (UniqueName: \"kubernetes.io/projected/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a-kube-api-access-mfdp9\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.019653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.019618 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:22:41.020195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.020160 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" podUID="0ab71b03-c225-41a7-9fa0-291123c90b86" containerName="maas-api" containerID="cri-o://9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4" gracePeriod=30 Oct 31 01:22:41.285440 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.285404 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:22:41.395677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.395643 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:22:41.451308 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451341 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451500 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451528 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451566 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451646 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451594 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451648 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451754 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.451856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.451793 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\") pod \"261e6357-4d20-4a20-be52-3e0f630fed32\" (UID: \"261e6357-4d20-4a20-be52-3e0f630fed32\") " Oct 31 01:22:41.452580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.452357 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:41.452745 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.452716 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:41.452895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.452853 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:41.453457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.453421 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:41.453562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.453461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:41.454971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.454944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:41.455596 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.455570 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:41.456144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.456117 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:41.456932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.456907 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt" pod "261e6357-4d20-4a20-be52-3e0f630fed32" (UID: "261e6357-4d20-4a20-be52-3e0f630fed32"). InnerVolumeSpecName "kube-api-access-l4fdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:41.532516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.532352 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" event={"ID":"e0372592-631a-42ed-87c3-a2b76bd1d84a","Type":"ContainerStarted","Data":"a88f2adc45e45e31bb1271715e318f82ed7cc506a03b8588d498885531dc415a"} Oct 31 01:22:41.532516 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.532423 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" event={"ID":"e0372592-631a-42ed-87c3-a2b76bd1d84a","Type":"ContainerStarted","Data":"17105ffce231a66208c829b8aba50fef0d2eae46f14ec934291ae714dbdf46d9"} Oct 31 01:22:41.534462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.534431 5011 generic.go:356] "Generic (PLEG): container finished" podID="261e6357-4d20-4a20-be52-3e0f630fed32" containerID="a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2" exitCode=0 Oct 31 01:22:41.534610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.534470 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" Oct 31 01:22:41.534610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.534508 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerDied","Data":"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2"} Oct 31 01:22:41.534610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.534555 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s" event={"ID":"261e6357-4d20-4a20-be52-3e0f630fed32","Type":"ContainerDied","Data":"48873a5646cd7519242ad7bf13c092fb2b4e3345a0050dc0dee367fb2e538c4a"} Oct 31 01:22:41.534610 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.534592 5011 scope.go:122] "RemoveContainer" containerID="a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2" Oct 31 01:22:41.537094 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.537070 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" event={"ID":"ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a","Type":"ContainerDied","Data":"5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560"} Oct 31 01:22:41.537246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.537116 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8b0af2a8086e897f9d18750c8d5a28fe1ed51e2ae73760e4b743bd3deb0560" Oct 31 01:22:41.537246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.537194 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr" Oct 31 01:22:41.540218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.539313 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" event={"ID":"04e45ad7-1498-4d79-96ab-139941c2dfa9","Type":"ContainerDied","Data":"79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45"} Oct 31 01:22:41.540218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.539360 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79969e0892f2ea7d935a1589fa621de7e0cb64b805a871eceae8a44977853d45" Oct 31 01:22:41.540218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.539435 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns" Oct 31 01:22:41.547112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.547082 5011 generic.go:356] "Generic (PLEG): container finished" podID="0ab71b03-c225-41a7-9fa0-291123c90b86" containerID="9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4" exitCode=0 Oct 31 01:22:41.547751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.547718 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" event={"ID":"0ab71b03-c225-41a7-9fa0-291123c90b86","Type":"ContainerDied","Data":"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4"} Oct 31 01:22:41.547850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.547770 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" event={"ID":"0ab71b03-c225-41a7-9fa0-291123c90b86","Type":"ContainerDied","Data":"4e63921435dcdbf5c1b639b43e199344cae88b4111d7f4f479f49260fbc1704a"} Oct 31 01:22:41.547926 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.547853 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq" Oct 31 01:22:41.556539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.556512 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\") pod \"0ab71b03-c225-41a7-9fa0-291123c90b86\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " Oct 31 01:22:41.556656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.556569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\" (UniqueName: \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\") pod \"0ab71b03-c225-41a7-9fa0-291123c90b86\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " Oct 31 01:22:41.556746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.556659 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\") pod \"0ab71b03-c225-41a7-9fa0-291123c90b86\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " Oct 31 01:22:41.556746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.556713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\") pod \"0ab71b03-c225-41a7-9fa0-291123c90b86\" (UID: \"0ab71b03-c225-41a7-9fa0-291123c90b86\") " Oct 31 01:22:41.557190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557157 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.557190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557184 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557202 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557234 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557245 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261e6357-4d20-4a20-be52-3e0f630fed32-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557257 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557269 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261e6357-4d20-4a20-be52-3e0f630fed32-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557282 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261e6357-4d20-4a20-be52-3e0f630fed32-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557294 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fdt\" (UniqueName: \"kubernetes.io/projected/261e6357-4d20-4a20-be52-3e0f630fed32-kube-api-access-l4fdt\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.568518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.557596 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca" pod "0ab71b03-c225-41a7-9fa0-291123c90b86" (UID: "0ab71b03-c225-41a7-9fa0-291123c90b86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:41.593260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.591646 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z" pod "0ab71b03-c225-41a7-9fa0-291123c90b86" (UID: "0ab71b03-c225-41a7-9fa0-291123c90b86"). InnerVolumeSpecName "kube-api-access-vzx6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:41.593260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.591726 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls" pod "0ab71b03-c225-41a7-9fa0-291123c90b86" (UID: "0ab71b03-c225-41a7-9fa0-291123c90b86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:41.608348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.608311 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs" pod "0ab71b03-c225-41a7-9fa0-291123c90b86" (UID: "0ab71b03-c225-41a7-9fa0-291123c90b86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:41.653247 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.643966 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" podStartSLOduration=2.643942889 podStartE2EDuration="2.643942889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:41.590140971 +0000 UTC m=+3830.039228611" watchObservedRunningTime="2026-10-31 01:22:41.643942889 +0000 UTC m=+3830.093030529" Oct 31 01:22:41.660483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.660361 5011 scope.go:122] "RemoveContainer" containerID="06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4" Oct 31 01:22:41.662762 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.662740 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-maas-api-tls\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.662906 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.662892 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab71b03-c225-41a7-9fa0-291123c90b86-openshift-service-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.663011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.662998 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab71b03-c225-41a7-9fa0-291123c90b86-metrics-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.663145 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.663131 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzx6z\" (UniqueName: \"kubernetes.io/projected/0ab71b03-c225-41a7-9fa0-291123c90b86-kube-api-access-vzx6z\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:41.679023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.678997 5011 scope.go:122] "RemoveContainer" containerID="a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2" Oct 31 01:22:41.679411 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:41.679392 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2\": container with ID starting with a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2 not found: ID does not exist" containerID="a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2" Oct 31 01:22:41.679471 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.679422 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2"} err="failed to get container status \"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2\": rpc error: code = NotFound desc = could not find container \"a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2\": container with ID starting with a91d942f73cf94d8abc1db63ee36bb6695937b052b5c2590ad7efa1d080f03a2 not found: ID does not exist" Oct 31 01:22:41.679471 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.679439 5011 scope.go:122] "RemoveContainer" containerID="06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4" Oct 31 01:22:41.679726 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:41.679710 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4\": container with ID starting with 06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4 not found: ID does not exist" containerID="06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4" Oct 31 01:22:41.679780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.679731 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4"} err="failed to get container status \"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4\": rpc error: code = NotFound desc = could not find container \"06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4\": container with ID starting with 06bd65643b259637025e868a3ab5298f40e7e864702dbe56a36e6ebb84e92ef4 not found: ID does not exist" Oct 31 01:22:41.679780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.679745 5011 scope.go:122] "RemoveContainer" containerID="9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4" Oct 31 01:22:41.690636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.690618 5011 scope.go:122] "RemoveContainer" containerID="9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4" Oct 31 01:22:41.690931 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:41.690912 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4\": container with ID starting with 9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4 not found: ID does not exist" containerID="9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4" Oct 31 01:22:41.690981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.690937 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4"} err="failed to get container status \"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4\": rpc error: code = NotFound desc = could not find container \"9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4\": container with ID starting with 9acdd435888978b32d09210f9792f7f7c1b43e05b0b60b627e05c39667f604f4 not found: ID does not exist" Oct 31 01:22:41.714704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.714679 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:22:41.722529 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.722500 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d10721-openshift-default-bf897765f-gt67s"] Oct 31 01:22:41.858681 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.858649 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg"] Oct 31 01:22:41.862070 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.862048 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68696-l7xdg"] Oct 31 01:22:41.893075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.893029 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:22:41.896361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:41.896320 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68696-6848b8bc9b-5r2zq"] Oct 31 01:22:42.560529 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:42.560493 5011 generic.go:356] "Generic (PLEG): container finished" podID="e0372592-631a-42ed-87c3-a2b76bd1d84a" containerID="a88f2adc45e45e31bb1271715e318f82ed7cc506a03b8588d498885531dc415a" exitCode=0 Oct 31 01:22:42.560913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:42.560571 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" event={"ID":"e0372592-631a-42ed-87c3-a2b76bd1d84a","Type":"ContainerDied","Data":"a88f2adc45e45e31bb1271715e318f82ed7cc506a03b8588d498885531dc415a"} Oct 31 01:22:43.686073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.686040 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab71b03-c225-41a7-9fa0-291123c90b86" path="/var/lib/kubelet/pods/0ab71b03-c225-41a7-9fa0-291123c90b86/volumes" Oct 31 01:22:43.686553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.686485 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e14973c-d6d7-4605-8e02-0523844b8a5f" path="/var/lib/kubelet/pods/0e14973c-d6d7-4605-8e02-0523844b8a5f/volumes" Oct 31 01:22:43.686944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.686927 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" path="/var/lib/kubelet/pods/261e6357-4d20-4a20-be52-3e0f630fed32/volumes" Oct 31 01:22:43.741544 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.741520 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:43.891826 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.891706 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\") pod \"e0372592-631a-42ed-87c3-a2b76bd1d84a\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " Oct 31 01:22:43.895228 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.892158 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\") pod \"e0372592-631a-42ed-87c3-a2b76bd1d84a\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " Oct 31 01:22:43.915566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.915538 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\" (UniqueName: \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\") pod \"e0372592-631a-42ed-87c3-a2b76bd1d84a\" (UID: \"e0372592-631a-42ed-87c3-a2b76bd1d84a\") " Oct 31 01:22:43.924319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.893625 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca" pod "e0372592-631a-42ed-87c3-a2b76bd1d84a" (UID: "e0372592-631a-42ed-87c3-a2b76bd1d84a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:43.924319 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.914089 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca" pod "e0372592-631a-42ed-87c3-a2b76bd1d84a" (UID: "e0372592-631a-42ed-87c3-a2b76bd1d84a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:43.926231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:43.925196 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t" pod "e0372592-631a-42ed-87c3-a2b76bd1d84a" (UID: "e0372592-631a-42ed-87c3-a2b76bd1d84a"). InnerVolumeSpecName "kube-api-access-pqn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:44.017307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.017260 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:44.017307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.017306 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqn7t\" (UniqueName: \"kubernetes.io/projected/e0372592-631a-42ed-87c3-a2b76bd1d84a-kube-api-access-pqn7t\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:44.017564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.017322 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0372592-631a-42ed-87c3-a2b76bd1d84a-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:44.576282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.576250 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" Oct 31 01:22:44.576507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.576242 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth" event={"ID":"e0372592-631a-42ed-87c3-a2b76bd1d84a","Type":"ContainerDied","Data":"17105ffce231a66208c829b8aba50fef0d2eae46f14ec934291ae714dbdf46d9"} Oct 31 01:22:44.576507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:44.576417 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17105ffce231a66208c829b8aba50fef0d2eae46f14ec934291ae714dbdf46d9" Oct 31 01:22:47.499477 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.499426 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:22:47.500905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.500873 5011 cpu_manager.go:414] "containerMap: removing container" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.501080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501065 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.501188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501177 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0ab71b03-c225-41a7-9fa0-291123c90b86" containerName="maas-api" Oct 31 01:22:47.501307 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501292 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab71b03-c225-41a7-9fa0-291123c90b86" containerName="maas-api" Oct 31 01:22:47.501417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501394 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e0372592-631a-42ed-87c3-a2b76bd1d84a" containerName="revoke-keys" Oct 31 01:22:47.501417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501412 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0372592-631a-42ed-87c3-a2b76bd1d84a" containerName="revoke-keys" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501426 5011 cpu_manager.go:414] "containerMap: removing container" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501433 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501447 5011 cpu_manager.go:414] "containerMap: removing container" podUID="04e45ad7-1498-4d79-96ab-139941c2dfa9" containerName="revoke-keys" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501456 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e45ad7-1498-4d79-96ab-139941c2dfa9" containerName="revoke-keys" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501488 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" containerName="revoke-keys" Oct 31 01:22:47.501622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501496 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" containerName="revoke-keys" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501658 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" containerName="revoke-keys" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501679 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501687 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab71b03-c225-41a7-9fa0-291123c90b86" containerName="maas-api" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501696 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0372592-631a-42ed-87c3-a2b76bd1d84a" containerName="revoke-keys" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501707 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="261e6357-4d20-4a20-be52-3e0f630fed32" containerName="istio-proxy" Oct 31 01:22:47.502006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.501715 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e45ad7-1498-4d79-96ab-139941c2dfa9" containerName="revoke-keys" Oct 31 01:22:47.511522 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.511492 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.515407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.515380 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:22:47.520300 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.520265 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.521224 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.521189 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:22:47.544097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.544069 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:22:47.554324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.554446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554336 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554360 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wj6\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.554570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554485 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554570 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554530 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.554708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554607 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.554708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554673 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554708 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554701 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.554873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554741 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554765 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554794 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.554873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.554821 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.555107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.555107 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555086 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.555339 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555140 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.555339 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555175 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.555339 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555226 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spl5s\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.555339 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.555268 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.656789 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656752 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656808 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656837 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656870 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656902 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656928 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.656982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.656982 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657008 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657089 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657141 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657175 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spl5s\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657372 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657413 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.657427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657426 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wj6\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657613 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657783 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657895 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.657997 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.658128 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658546 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.658488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.658679 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.658640 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.659988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.659960 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.660097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.660045 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.660097 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.660064 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.660563 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.660544 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.680428 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.680405 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.680735 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.680711 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wj6\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.680882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.680859 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spl5s\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.680963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.680938 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:47.832297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.832265 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:47.841848 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:47.841822 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:48.171640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.171487 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:22:48.171640 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:48.171612 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4828e3_763a_4fd5_bd5d_74b3c5bb8c04.slice/crio-bae8604185e639057a29763b16437d2c890840dd48e170be1c275d9ab45dad75 WatchSource:0}: Error finding container bae8604185e639057a29763b16437d2c890840dd48e170be1c275d9ab45dad75: Status 404 returned error can't find the container with id bae8604185e639057a29763b16437d2c890840dd48e170be1c275d9ab45dad75 Oct 31 01:22:48.174413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.174377 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.174542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.174440 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.174542 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.174476 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.175788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.175761 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:22:48.196700 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:22:48.196579 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842bb9ce_48b0_4766_aa72_1507a421e8ef.slice/crio-59782096e5bf7e651eb38ab06bf41010c3bc9c6f84a15965f35656deb7e557dc WatchSource:0}: Error finding container 59782096e5bf7e651eb38ab06bf41010c3bc9c6f84a15965f35656deb7e557dc: Status 404 returned error can't find the container with id 59782096e5bf7e651eb38ab06bf41010c3bc9c6f84a15965f35656deb7e557dc Oct 31 01:22:48.199751 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.199718 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.199856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.199782 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.199856 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.199810 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:22:48.609600 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.609550 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" event={"ID":"842bb9ce-48b0-4766-aa72-1507a421e8ef","Type":"ContainerStarted","Data":"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57"} Oct 31 01:22:48.610120 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.609651 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" event={"ID":"842bb9ce-48b0-4766-aa72-1507a421e8ef","Type":"ContainerStarted","Data":"59782096e5bf7e651eb38ab06bf41010c3bc9c6f84a15965f35656deb7e557dc"} Oct 31 01:22:48.611630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.611596 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" event={"ID":"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04","Type":"ContainerStarted","Data":"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820"} Oct 31 01:22:48.611786 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.611654 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" event={"ID":"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04","Type":"ContainerStarted","Data":"bae8604185e639057a29763b16437d2c890840dd48e170be1c275d9ab45dad75"} Oct 31 01:22:48.657991 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.657934 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podStartSLOduration=1.6579190000000001 podStartE2EDuration="1.657919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.654607725 +0000 UTC m=+3837.103695365" watchObservedRunningTime="2026-10-31 01:22:48.657919 +0000 UTC m=+3837.107006639" Oct 31 01:22:48.703847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.703789 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podStartSLOduration=1.703775467 podStartE2EDuration="1.703775467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.700346352 +0000 UTC m=+3837.149434014" watchObservedRunningTime="2026-10-31 01:22:48.703775467 +0000 UTC m=+3837.152863106" Oct 31 01:22:48.835350 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.835307 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:48.843343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:48.843302 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:49.598367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.595906 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:22:49.598367 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.596174 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" containerID="cri-o://aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987" gracePeriod=30 Oct 31 01:22:49.843233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.839401 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 31 01:22:49.843233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.839469 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 31 01:22:49.867577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.867486 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 31 01:22:49.867851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:49.867821 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 31 01:22:50.151285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.139888 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52"] Oct 31 01:22:50.156568 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.152418 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.178274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.169321 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52"] Oct 31 01:22:50.178274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.169654 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:22:50.178274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.169820 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:22:50.241222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.241167 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:50.241645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.241623 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:22:50.282369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.269793 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.282369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.269949 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.282369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.269987 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpspj\" (UniqueName: \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.372279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.371146 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.372279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.371360 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.372279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.371404 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpspj\" (UniqueName: \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.392410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.389343 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.392410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.389728 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.406285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.405425 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpspj\" (UniqueName: \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.509282 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.499087 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:50.843241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.842650 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 31 01:22:50.843241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.842713 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 31 01:22:50.870239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.864722 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 31 01:22:50.870239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:50.864785 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 31 01:22:51.058632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.053234 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52"] Oct 31 01:22:51.800686 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.800653 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" event={"ID":"134c516e-4649-4479-afd7-cdacdb3421f2","Type":"ContainerStarted","Data":"af279d498c54d5a51853d582532400b2cc69a20973406b43d207497b81d73529"} Oct 31 01:22:51.800857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.800703 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" event={"ID":"134c516e-4649-4479-afd7-cdacdb3421f2","Type":"ContainerStarted","Data":"607b87708ec134f13636e773dc631532cdc41af5d1dec7c50865e176762caec9"} Oct 31 01:22:51.834458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.833599 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 31 01:22:51.834458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.833656 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 31 01:22:51.842606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.842561 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 31 01:22:51.842852 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.842821 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 31 01:22:51.883941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:51.883873 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" podStartSLOduration=1.883852874 podStartE2EDuration="1.883852874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:51.857524643 +0000 UTC m=+3840.306612285" watchObservedRunningTime="2026-10-31 01:22:51.883852874 +0000 UTC m=+3840.332940514" Oct 31 01:22:52.584278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.583992 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:22:52.584278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.584053 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:22:52.878499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.876452 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 31 01:22:52.878499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.876519 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 31 01:22:52.878499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.878378 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 31 01:22:52.878499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.878429 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 31 01:22:52.881838 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.881804 5011 generic.go:356] "Generic (PLEG): container finished" podID="134c516e-4649-4479-afd7-cdacdb3421f2" containerID="af279d498c54d5a51853d582532400b2cc69a20973406b43d207497b81d73529" exitCode=0 Oct 31 01:22:52.881996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:52.881843 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" event={"ID":"134c516e-4649-4479-afd7-cdacdb3421f2","Type":"ContainerDied","Data":"af279d498c54d5a51853d582532400b2cc69a20973406b43d207497b81d73529"} Oct 31 01:22:53.092179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.091444 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94"] Oct 31 01:22:53.096486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.096457 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33443aec-clm94"] Oct 31 01:22:53.689318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.689278 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fa412f-d98b-4bd6-ba9c-f53428348476" path="/var/lib/kubelet/pods/e4fa412f-d98b-4bd6-ba9c-f53428348476/volumes" Oct 31 01:22:53.836240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.835621 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 31 01:22:53.836240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.835700 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 31 01:22:53.866232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.849333 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 31 01:22:53.866232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:53.849406 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 31 01:22:54.213736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.213708 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:54.396161 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.396106 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\") pod \"134c516e-4649-4479-afd7-cdacdb3421f2\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " Oct 31 01:22:54.396396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.396222 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\") pod \"134c516e-4649-4479-afd7-cdacdb3421f2\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " Oct 31 01:22:54.396396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.396272 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\" (UniqueName: \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\") pod \"134c516e-4649-4479-afd7-cdacdb3421f2\" (UID: \"134c516e-4649-4479-afd7-cdacdb3421f2\") " Oct 31 01:22:54.396631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.396600 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca" pod "134c516e-4649-4479-afd7-cdacdb3421f2" (UID: "134c516e-4649-4479-afd7-cdacdb3421f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:54.396783 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.396753 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.399003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.398970 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca" pod "134c516e-4649-4479-afd7-cdacdb3421f2" (UID: "134c516e-4649-4479-afd7-cdacdb3421f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:54.399103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.398973 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj" pod "134c516e-4649-4479-afd7-cdacdb3421f2" (UID: "134c516e-4649-4479-afd7-cdacdb3421f2"). InnerVolumeSpecName "kube-api-access-kpspj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:54.497851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.497808 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpspj\" (UniqueName: \"kubernetes.io/projected/134c516e-4649-4479-afd7-cdacdb3421f2-kube-api-access-kpspj\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.497851 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.497849 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c516e-4649-4479-afd7-cdacdb3421f2-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.840621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.838978 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:54.840621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.840180 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:54.840979 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.840953 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:22:54.847131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.847107 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:54.847671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.847602 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:54.848507 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.848461 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:22:54.868677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.868648 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:22:54.902776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902752 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.902897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902836 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.902897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902859 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.902897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902885 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902944 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.902989 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903082 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903131 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903166 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:54.903332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903176 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\") pod \"38024f7b-d715-44ce-a5f2-363d5473b559\" (UID: \"38024f7b-d715-44ce-a5f2-363d5473b559\") " Oct 31 01:22:54.903442 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903401 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:54.903525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903502 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:54.903604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.903528 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:54.904137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.904107 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.904137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.904134 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.904310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.904149 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.904310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.904162 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38024f7b-d715-44ce-a5f2-363d5473b559-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:54.904388 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.904337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:54.905469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.905449 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:54.905713 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.905691 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:54.905969 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.905950 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "kube-api-access-qh88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:54.906063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.906041 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token" pod "38024f7b-d715-44ce-a5f2-363d5473b559" (UID: "38024f7b-d715-44ce-a5f2-363d5473b559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:54.917110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.917087 5011 generic.go:356] "Generic (PLEG): container finished" podID="38024f7b-d715-44ce-a5f2-363d5473b559" containerID="aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987" exitCode=0 Oct 31 01:22:54.917264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.917142 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" Oct 31 01:22:54.917264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.917171 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerDied","Data":"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987"} Oct 31 01:22:54.917264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.917246 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf" event={"ID":"38024f7b-d715-44ce-a5f2-363d5473b559","Type":"ContainerDied","Data":"f0d7eb392b9395668782ce9957cfa3a256f31fc7372408642d0bbc633cdc7dda"} Oct 31 01:22:54.917449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.917278 5011 scope.go:122] "RemoveContainer" containerID="aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987" Oct 31 01:22:54.918875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.918857 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" Oct 31 01:22:54.918980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.918933 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52" event={"ID":"134c516e-4649-4479-afd7-cdacdb3421f2","Type":"ContainerDied","Data":"607b87708ec134f13636e773dc631532cdc41af5d1dec7c50865e176762caec9"} Oct 31 01:22:54.919042 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.918977 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607b87708ec134f13636e773dc631532cdc41af5d1dec7c50865e176762caec9" Oct 31 01:22:54.930779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.930760 5011 scope.go:122] "RemoveContainer" containerID="d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb" Oct 31 01:22:54.943382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.943357 5011 scope.go:122] "RemoveContainer" containerID="aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987" Oct 31 01:22:54.943679 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:54.943654 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987\": container with ID starting with aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987 not found: ID does not exist" containerID="aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987" Oct 31 01:22:54.943756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.943692 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987"} err="failed to get container status \"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987\": rpc error: code = NotFound desc = could not find container \"aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987\": container with ID starting with aeb2d5b740bb5b798d549fe5160b48960270f487accbaf9804f73e83177a3987 not found: ID does not exist" Oct 31 01:22:54.943756 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.943716 5011 scope.go:122] "RemoveContainer" containerID="d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb" Oct 31 01:22:54.943987 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:22:54.943968 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb\": container with ID starting with d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb not found: ID does not exist" containerID="d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb" Oct 31 01:22:54.944061 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:54.943993 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb"} err="failed to get container status \"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb\": rpc error: code = NotFound desc = could not find container \"d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb\": container with ID starting with d07f2dbc6f9354f76024d0e06c3e438be3eed833a789014b7b4afae049ef7cfb not found: ID does not exist" Oct 31 01:22:55.006006 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.005922 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh88x\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-kube-api-access-qh88x\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.006185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.006113 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.006185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.006138 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38024f7b-d715-44ce-a5f2-363d5473b559-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.006185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.006152 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38024f7b-d715-44ce-a5f2-363d5473b559-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.006185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.006164 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38024f7b-d715-44ce-a5f2-363d5473b559-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.116876 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.116835 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:22:55.127022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.126993 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d68696-openshift-default-6dd844fcf-t84vf"] Oct 31 01:22:55.277508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.277429 5011 scope.go:122] "RemoveContainer" containerID="bf5ae48a62e89c0840b288e0cd4a3cc4e9e4b4d6add77f648b892be0dfd30170" Oct 31 01:22:55.767887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:55.767846 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" path="/var/lib/kubelet/pods/38024f7b-d715-44ce-a5f2-363d5473b559/volumes" Oct 31 01:22:56.049821 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:56.049735 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw"] Oct 31 01:22:56.055566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:56.055538 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ada0c1-2c5jw"] Oct 31 01:22:57.684548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:57.684508 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3d33b6-2ecb-4633-95a7-9456e25682f7" path="/var/lib/kubelet/pods/6a3d33b6-2ecb-4633-95a7-9456e25682f7/volumes" Oct 31 01:22:57.804299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:57.804258 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:22:58.370122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:58.369903 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj"] Oct 31 01:22:58.379106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:58.379074 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be301b-4fdhj"] Oct 31 01:22:59.682864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:22:59.682826 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4399fc62-7861-44de-9576-9e71ce1d252d" path="/var/lib/kubelet/pods/4399fc62-7861-44de-9576-9e71ce1d252d/volumes" Oct 31 01:23:01.041386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:01.041346 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4"] Oct 31 01:23:01.051386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:01.051352 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59346ee6-vvsp4"] Oct 31 01:23:01.683908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:01.683870 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580ea84d-5430-4a21-9d81-8ada37729dc4" path="/var/lib/kubelet/pods/580ea84d-5430-4a21-9d81-8ada37729dc4/volumes" Oct 31 01:23:01.948814 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:01.948718 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:23:02.537258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.537218 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:02.537996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.537975 5011 cpu_manager.go:414] "containerMap: removing container" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.537998 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538018 5011 cpu_manager.go:414] "containerMap: removing container" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538054 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538027 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538064 5011 cpu_manager.go:414] "containerMap: removing container" podUID="134c516e-4649-4479-afd7-cdacdb3421f2" containerName="revoke-keys" Oct 31 01:23:02.538182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538074 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134c516e-4649-4479-afd7-cdacdb3421f2" containerName="revoke-keys" Oct 31 01:23:02.538295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538240 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538255 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="38024f7b-d715-44ce-a5f2-363d5473b559" containerName="istio-proxy" Oct 31 01:23:02.538295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.538273 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="134c516e-4649-4479-afd7-cdacdb3421f2" containerName="revoke-keys" Oct 31 01:23:02.543717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.543694 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.571248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.571185 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:02.622824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.622791 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.622824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.622826 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hw2\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.622853 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.622962 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623079 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.623016 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.623142 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.623291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.623325 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.623407 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.623382 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.724671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.724864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724705 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.724864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724756 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.724864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724829 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724860 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.724896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725018 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725056 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hw2\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725097 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725380 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725433 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.725933 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.725907 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.728013 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.727988 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.728134 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.728027 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.749253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.749191 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.749390 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.749354 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hw2\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:02.874309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:02.874275 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:03.210073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:03.210037 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:03.210521 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:03.210498 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dd24a0_8686_42ed_9dd5_18adfe9073e3.slice/crio-fd6e2ab0528d6b44fea2cf65b6c30dcf93b5df55be32ec6531abe8f6baa649c6 WatchSource:0}: Error finding container fd6e2ab0528d6b44fea2cf65b6c30dcf93b5df55be32ec6531abe8f6baa649c6: Status 404 returned error can't find the container with id fd6e2ab0528d6b44fea2cf65b6c30dcf93b5df55be32ec6531abe8f6baa649c6 Oct 31 01:23:03.213055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:03.213023 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:23:03.213136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:03.213090 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:23:03.213136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:03.213119 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:23:04.002655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.002611 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" event={"ID":"47dd24a0-8686-42ed-9dd5-18adfe9073e3","Type":"ContainerStarted","Data":"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff"} Oct 31 01:23:04.003060 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.002690 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" event={"ID":"47dd24a0-8686-42ed-9dd5-18adfe9073e3","Type":"ContainerStarted","Data":"fd6e2ab0528d6b44fea2cf65b6c30dcf93b5df55be32ec6531abe8f6baa649c6"} Oct 31 01:23:04.062109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.062078 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g"] Oct 31 01:23:04.067629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.067591 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b727ecf5-rcw4g"] Oct 31 01:23:04.091101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.091041 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podStartSLOduration=2.09102292 podStartE2EDuration="2.09102292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:04.089799288 +0000 UTC m=+3852.538886928" watchObservedRunningTime="2026-10-31 01:23:04.09102292 +0000 UTC m=+3852.540110560" Oct 31 01:23:04.277755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.277662 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:23:04.277921 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.277897 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" containerID="cri-o://dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9" gracePeriod=30 Oct 31 01:23:04.875325 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.875276 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:04.876862 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.876837 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body= Oct 31 01:23:04.876928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:04.876890 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" Oct 31 01:23:05.010044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.010011 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29"] Oct 31 01:23:05.014179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.014152 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.053510 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.053476 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:23:05.053797 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.053777 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:05.054024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.054007 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:23:05.054156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.054142 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:05.055122 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.055086 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29"] Oct 31 01:23:05.179776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.179694 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.179929 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.179792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplkz\" (UniqueName: \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.180035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.180008 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.281435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.281402 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.281634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.281490 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.281634 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.281513 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplkz\" (UniqueName: \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.282080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.282055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.283909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.283886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.305639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.305611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplkz\" (UniqueName: \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\") pod \"maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.364545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.364513 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:05.682812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.682778 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61758a54-65dd-49e5-93c7-ff89f2f1b900" path="/var/lib/kubelet/pods/61758a54-65dd-49e5-93c7-ff89f2f1b900/volumes" Oct 31 01:23:05.705034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.705007 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29"] Oct 31 01:23:05.707185 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:05.707156 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1a8807_acba_45d1_9e16_2c0090eb525c.slice/crio-5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706 WatchSource:0}: Error finding container 5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706: Status 404 returned error can't find the container with id 5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706 Oct 31 01:23:05.875286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.875251 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body= Oct 31 01:23:05.875466 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:05.875309 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" Oct 31 01:23:06.033278 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:06.030663 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" event={"ID":"fe1a8807-acba-45d1-9e16-2c0090eb525c","Type":"ContainerStarted","Data":"5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706"} Oct 31 01:23:06.875556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:06.875515 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body= Oct 31 01:23:06.875728 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:06.875568 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" Oct 31 01:23:07.050443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:07.050353 5011 generic.go:356] "Generic (PLEG): container finished" podID="fe1a8807-acba-45d1-9e16-2c0090eb525c" containerID="d0b6cc9cc8a56b33b079ed443f72cc22fb241236facdbf775d22d4d1c5fc8d2d" exitCode=0 Oct 31 01:23:07.050880 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:07.050435 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" event={"ID":"fe1a8807-acba-45d1-9e16-2c0090eb525c","Type":"ContainerDied","Data":"d0b6cc9cc8a56b33b079ed443f72cc22fb241236facdbf775d22d4d1c5fc8d2d"} Oct 31 01:23:07.874882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:07.874836 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body= Oct 31 01:23:07.875052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:07.874901 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" Oct 31 01:23:08.277166 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.277138 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:08.349657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.349613 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\" (UniqueName: \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\") pod \"fe1a8807-acba-45d1-9e16-2c0090eb525c\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " Oct 31 01:23:08.349835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.349784 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\") pod \"fe1a8807-acba-45d1-9e16-2c0090eb525c\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " Oct 31 01:23:08.349880 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.349861 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\") pod \"fe1a8807-acba-45d1-9e16-2c0090eb525c\" (UID: \"fe1a8807-acba-45d1-9e16-2c0090eb525c\") " Oct 31 01:23:08.350403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.350372 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca" pod "fe1a8807-acba-45d1-9e16-2c0090eb525c" (UID: "fe1a8807-acba-45d1-9e16-2c0090eb525c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:08.350824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.350795 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:08.353505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.353476 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz" pod "fe1a8807-acba-45d1-9e16-2c0090eb525c" (UID: "fe1a8807-acba-45d1-9e16-2c0090eb525c"). InnerVolumeSpecName "kube-api-access-wplkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:08.353603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.353527 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca" pod "fe1a8807-acba-45d1-9e16-2c0090eb525c" (UID: "fe1a8807-acba-45d1-9e16-2c0090eb525c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:08.451685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.451598 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplkz\" (UniqueName: \"kubernetes.io/projected/fe1a8807-acba-45d1-9e16-2c0090eb525c-kube-api-access-wplkz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:08.451685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.451633 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1a8807-acba-45d1-9e16-2c0090eb525c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:08.875346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.875310 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body= Oct 31 01:23:08.875557 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:08.875363 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" Oct 31 01:23:09.067642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.067606 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" Oct 31 01:23:09.067792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.067640 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29" event={"ID":"fe1a8807-acba-45d1-9e16-2c0090eb525c","Type":"ContainerDied","Data":"5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706"} Oct 31 01:23:09.067792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.067697 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7faa4bd6a034140bbbd3377d855d396dbfcdea01631d3d467de5b418fa9706" Oct 31 01:23:09.730050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.730015 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:23:09.732661 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.731851 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 01:23:09.740925 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.740900 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 01:23:09.759857 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.759823 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 01:23:09.876878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.876838 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.876878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.876879 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.876896 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.876927 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.876975 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.877054 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.877080 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.877142 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.877430 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.877164 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\") pod \"74407c2a-2b5d-4185-955e-b47c4e73d766\" (UID: \"74407c2a-2b5d-4185-955e-b47c4e73d766\") " Oct 31 01:23:09.878714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.878680 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.878907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.878886 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.879091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.879071 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.879861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.879828 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.879964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.879856 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "kube-api-access-lrwd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:09.880166 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.880143 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:09.880651 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.880625 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:09.881202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.881181 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:09.881352 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.881330 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:09.881714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.881695 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:09.882405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.882382 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data" pod "74407c2a-2b5d-4185-955e-b47c4e73d766" (UID: "74407c2a-2b5d-4185-955e-b47c4e73d766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.882647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.882625 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:10.004318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:09.986335 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.004575 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.004549 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrwd8\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-kube-api-access-lrwd8\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.004696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.004681 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.014289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.014255 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.036234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.025254 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.036234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.025287 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.036234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.025301 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74407c2a-2b5d-4185-955e-b47c4e73d766-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.036234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.025318 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74407c2a-2b5d-4185-955e-b47c4e73d766-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.036234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.025332 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74407c2a-2b5d-4185-955e-b47c4e73d766-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:10.112998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.111166 5011 generic.go:356] "Generic (PLEG): container finished" podID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerID="dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9" exitCode=0 Oct 31 01:23:10.112998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.111374 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" Oct 31 01:23:10.112998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.112292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerDied","Data":"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9"} Oct 31 01:23:10.112998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.112345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b" event={"ID":"74407c2a-2b5d-4185-955e-b47c4e73d766","Type":"ContainerDied","Data":"ced8e60aacd8f723d4df8d118517ea90e8c661ed922b7e1707a56939ad15c03a"} Oct 31 01:23:10.112998 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.112379 5011 scope.go:122] "RemoveContainer" containerID="dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9" Oct 31 01:23:10.139109 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.139085 5011 scope.go:122] "RemoveContainer" containerID="8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.153388 5011 scope.go:122] "RemoveContainer" containerID="dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:10.153746 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9\": container with ID starting with dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9 not found: ID does not exist" containerID="dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.153785 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9"} err="failed to get container status \"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9\": rpc error: code = NotFound desc = could not find container \"dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9\": container with ID starting with dc28c22ad946e98606d69b0aaf05a93b4d09cd54a9d4498e28111bb87a74beb9 not found: ID does not exist" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.153801 5011 scope.go:122] "RemoveContainer" containerID="8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:10.154089 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a\": container with ID starting with 8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a not found: ID does not exist" containerID="8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a" Oct 31 01:23:10.154971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.154112 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a"} err="failed to get container status \"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a\": rpc error: code = NotFound desc = could not find container \"8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a\": container with ID starting with 8b7cdcee8dfffeaa7669ab9f34e49710251674d8d37035b33a5354b2f29a787a not found: ID does not exist" Oct 31 01:23:10.183601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.183564 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:23:10.191832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.191789 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-n474b"] Oct 31 01:23:10.294532 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.294443 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth"] Oct 31 01:23:10.297591 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:10.297562 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-114c95-59wth"] Oct 31 01:23:11.684017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:11.683977 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" path="/var/lib/kubelet/pods/74407c2a-2b5d-4185-955e-b47c4e73d766/volumes" Oct 31 01:23:11.684547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:11.684501 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0372592-631a-42ed-87c3-a2b76bd1d84a" path="/var/lib/kubelet/pods/e0372592-631a-42ed-87c3-a2b76bd1d84a/volumes" Oct 31 01:23:12.032066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:12.031982 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f"] Oct 31 01:23:12.044049 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:12.044022 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892ec1da-x4f4f"] Oct 31 01:23:13.682932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:13.682897 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee39e65-a8cd-49ca-96f6-8ee31baaf43b" path="/var/lib/kubelet/pods/fee39e65-a8cd-49ca-96f6-8ee31baaf43b/volumes" Oct 31 01:23:15.031744 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:15.031696 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj"] Oct 31 01:23:15.043770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:15.043737 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a67d8597-6bjgj"] Oct 31 01:23:15.683105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:15.683070 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881dfccb-42ae-4332-a940-c289aa970527" path="/var/lib/kubelet/pods/881dfccb-42ae-4332-a940-c289aa970527/volumes" Oct 31 01:23:17.946017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:17.945977 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52"] Oct 31 01:23:17.957828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:17.957796 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99a5c2-c8q52"] Oct 31 01:23:18.631714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.631670 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5"] Oct 31 01:23:18.632316 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632296 5011 cpu_manager.go:414] "containerMap: removing container" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:18.632443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632317 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:18.632443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632345 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a8807-acba-45d1-9e16-2c0090eb525c" containerName="revoke-keys" Oct 31 01:23:18.632443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632352 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a8807-acba-45d1-9e16-2c0090eb525c" containerName="revoke-keys" Oct 31 01:23:18.632611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632497 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:18.632611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.632515 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a8807-acba-45d1-9e16-2c0090eb525c" containerName="revoke-keys" Oct 31 01:23:18.637457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.637433 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.656318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.656296 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5"] Oct 31 01:23:18.663342 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.663316 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:23:18.663342 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.663323 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:23:18.687785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.687760 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:18.687898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.687840 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:18.723687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.723655 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.723808 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.723795 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.723882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.723834 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87b7g\" (UniqueName: \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.824883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.824849 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.825019 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.824963 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.825019 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.825014 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87b7g\" (UniqueName: \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.825748 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.825722 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.827346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.827318 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.848453 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.848418 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87b7g\" (UniqueName: \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:18.956031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:18.955950 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:19.261100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:19.261064 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5"] Oct 31 01:23:19.266410 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:19.266374 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6743fcc6_aa1e_4a7b_89e0_3414e93e8309.slice/crio-7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96 WatchSource:0}: Error finding container 7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96: Status 404 returned error can't find the container with id 7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96 Oct 31 01:23:19.682894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:19.682860 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134c516e-4649-4479-afd7-cdacdb3421f2" path="/var/lib/kubelet/pods/134c516e-4649-4479-afd7-cdacdb3421f2/volumes" Oct 31 01:23:20.190332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:20.190299 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" event={"ID":"6743fcc6-aa1e-4a7b-89e0-3414e93e8309","Type":"ContainerStarted","Data":"1669bc798e3982dd4f8cd4bd42bd870c1ef730e9c2a5a07356cf937219c4489f"} Oct 31 01:23:20.190811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:20.190359 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" event={"ID":"6743fcc6-aa1e-4a7b-89e0-3414e93e8309","Type":"ContainerStarted","Data":"7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96"} Oct 31 01:23:20.231973 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:20.231930 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" podStartSLOduration=2.231916136 podStartE2EDuration="2.231916136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:20.230756954 +0000 UTC m=+3868.679844593" watchObservedRunningTime="2026-10-31 01:23:20.231916136 +0000 UTC m=+3868.681003775" Oct 31 01:23:21.222233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:21.208656 5011 generic.go:356] "Generic (PLEG): container finished" podID="6743fcc6-aa1e-4a7b-89e0-3414e93e8309" containerID="1669bc798e3982dd4f8cd4bd42bd870c1ef730e9c2a5a07356cf937219c4489f" exitCode=0 Oct 31 01:23:21.222233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:21.208784 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" event={"ID":"6743fcc6-aa1e-4a7b-89e0-3414e93e8309","Type":"ContainerDied","Data":"1669bc798e3982dd4f8cd4bd42bd870c1ef730e9c2a5a07356cf937219c4489f"} Oct 31 01:23:22.358960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.358931 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:22.476234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.476131 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\") pod \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " Oct 31 01:23:22.476234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.476170 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\") pod \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " Oct 31 01:23:22.476483 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.476250 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\" (UniqueName: \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\") pod \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\" (UID: \"6743fcc6-aa1e-4a7b-89e0-3414e93e8309\") " Oct 31 01:23:22.476556 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.476503 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca" pod "6743fcc6-aa1e-4a7b-89e0-3414e93e8309" (UID: "6743fcc6-aa1e-4a7b-89e0-3414e93e8309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:22.478420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.478397 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca" pod "6743fcc6-aa1e-4a7b-89e0-3414e93e8309" (UID: "6743fcc6-aa1e-4a7b-89e0-3414e93e8309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:22.478508 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.478449 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g" pod "6743fcc6-aa1e-4a7b-89e0-3414e93e8309" (UID: "6743fcc6-aa1e-4a7b-89e0-3414e93e8309"). InnerVolumeSpecName "kube-api-access-87b7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:22.577893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.577864 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:22.577893 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.577894 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:22.578080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:22.577911 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87b7g\" (UniqueName: \"kubernetes.io/projected/6743fcc6-aa1e-4a7b-89e0-3414e93e8309-kube-api-access-87b7g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.039002 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.038965 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq"] Oct 31 01:23:23.051871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.051840 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5597e43-ckmbq"] Oct 31 01:23:23.226684 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.226652 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" Oct 31 01:23:23.226861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.226697 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5" event={"ID":"6743fcc6-aa1e-4a7b-89e0-3414e93e8309","Type":"ContainerDied","Data":"7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96"} Oct 31 01:23:23.226861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.226745 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7277a96a276f7efb0f7c83057fd34e75b94062065e53aa53e814605308659f96" Oct 31 01:23:23.683162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:23.683127 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0802494-f13e-41f6-adbc-ca03e491f689" path="/var/lib/kubelet/pods/a0802494-f13e-41f6-adbc-ca03e491f689/volumes" Oct 31 01:23:24.764063 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:24.764014 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" containerID="cri-o://2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf" gracePeriod=45 Oct 31 01:23:24.776157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:24.776082 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" containerID="cri-o://3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e" gracePeriod=45 Oct 31 01:23:24.789682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:24.789645 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" containerID="cri-o://41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f" gracePeriod=45 Oct 31 01:23:25.001790 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.001746 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:23:25.002069 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.002031 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" containerID="cri-o://ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec" gracePeriod=30 Oct 31 01:23:25.157538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.157158 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 01:23:25.160898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.160867 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 01:23:25.165125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.165099 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 01:23:25.280892 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.280723 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.281144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.281070 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.281335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.281314 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.281668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.281640 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\" (UniqueName: \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.281888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.281867 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\" (UniqueName: \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.282102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.282083 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.282335 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.282314 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.282593 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.282563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\" (UniqueName: \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.282781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.282759 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.282992 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.282970 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.283172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.283154 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.283376 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.283577 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.283824 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284053 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\") pod \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\" (UID: \"13cb12cf-6050-4d2a-be4b-01a159d5a3c6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284241 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284260 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284605 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\") pod \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\" (UID: \"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6\") " Oct 31 01:23:25.285252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.284854 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\") pod \"6173beed-90e9-49b3-82a8-6404a4d943c6\" (UID: \"6173beed-90e9-49b3-82a8-6404a4d943c6\") " Oct 31 01:23:25.285924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.285871 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:25.287835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.287111 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.291376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.289217 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.291376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.289662 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.291376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.290026 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.291376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.290373 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.291376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.290390 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:25.293400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.293366 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.294167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.293956 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "kube-api-access-rxgxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:25.294315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.294272 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "kube-api-access-8ck8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:25.294472 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.294452 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.295627 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.295604 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm" pod "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" (UID: "bdb915eb-5c6f-4f70-99a8-7a7bdac856a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.297497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.297475 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.306251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.305335 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "kube-api-access-tnmvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:25.309139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309024 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309157 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgxh\" (UniqueName: \"kubernetes.io/projected/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-kube-api-access-rxgxh\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309175 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309427 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309409 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309505 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309488 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309524 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309540 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309558 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309572 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309625 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309639 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173beed-90e9-49b3-82a8-6404a4d943c6-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309658 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309671 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309835 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.309952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.309850 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ck8l\" (UniqueName: \"kubernetes.io/projected/6173beed-90e9-49b3-82a8-6404a4d943c6-kube-api-access-8ck8l\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.326188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.324737 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm" pod "13cb12cf-6050-4d2a-be4b-01a159d5a3c6" (UID: "13cb12cf-6050-4d2a-be4b-01a159d5a3c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.326188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.325168 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmvf\" (UniqueName: \"kubernetes.io/projected/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-kube-api-access-tnmvf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.329726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.329627 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs" pod "6173beed-90e9-49b3-82a8-6404a4d943c6" (UID: "6173beed-90e9-49b3-82a8-6404a4d943c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:25.341368 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.341333 5011 generic.go:356] "Generic (PLEG): container finished" podID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerID="3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e" exitCode=0 Oct 31 01:23:25.341548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.341400 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" Oct 31 01:23:25.341548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.341510 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" event={"ID":"13cb12cf-6050-4d2a-be4b-01a159d5a3c6","Type":"ContainerDied","Data":"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e"} Oct 31 01:23:25.341676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.341552 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5" event={"ID":"13cb12cf-6050-4d2a-be4b-01a159d5a3c6","Type":"ContainerDied","Data":"2052b81022a2f30534ba03acfe9ba5afac9c426d50a06d101c62a3e68afd0edb"} Oct 31 01:23:25.341676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.341578 5011 scope.go:122] "RemoveContainer" containerID="3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e" Oct 31 01:23:25.343923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.343703 5011 generic.go:356] "Generic (PLEG): container finished" podID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerID="41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f" exitCode=0 Oct 31 01:23:25.343923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.343819 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" event={"ID":"6173beed-90e9-49b3-82a8-6404a4d943c6","Type":"ContainerDied","Data":"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f"} Oct 31 01:23:25.343923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.343851 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" event={"ID":"6173beed-90e9-49b3-82a8-6404a4d943c6","Type":"ContainerDied","Data":"5f7280428aa8a38bb0aa67fa63554c06422dc0cd27a209a04700c6b7a8199c5f"} Oct 31 01:23:25.344108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.343922 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5" Oct 31 01:23:25.345984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.345957 5011 generic.go:356] "Generic (PLEG): container finished" podID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerID="2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf" exitCode=0 Oct 31 01:23:25.346192 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.346169 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" event={"ID":"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6","Type":"ContainerDied","Data":"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf"} Oct 31 01:23:25.346361 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.346344 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" event={"ID":"bdb915eb-5c6f-4f70-99a8-7a7bdac856a6","Type":"ContainerDied","Data":"dcbb5f8341d9d70883aa592d6300fcadc0d665974adb6403905aef6166653ee9"} Oct 31 01:23:25.346606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.346587 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" Oct 31 01:23:25.358985 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.358962 5011 scope.go:122] "RemoveContainer" containerID="3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e" Oct 31 01:23:25.359527 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:25.359354 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e\": container with ID starting with 3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e not found: ID does not exist" containerID="3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e" Oct 31 01:23:25.359527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.359412 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e"} err="failed to get container status \"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e\": rpc error: code = NotFound desc = could not find container \"3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e\": container with ID starting with 3b5f625d5b7b2c4314fb4503fe6e32ad7cc464e1a4812892c864446933473c4e not found: ID does not exist" Oct 31 01:23:25.359527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.359438 5011 scope.go:122] "RemoveContainer" containerID="41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f" Oct 31 01:23:25.372291 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.372228 5011 scope.go:122] "RemoveContainer" containerID="41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f" Oct 31 01:23:25.373736 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:25.373579 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f\": container with ID starting with 41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f not found: ID does not exist" containerID="41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f" Oct 31 01:23:25.373736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.373618 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f"} err="failed to get container status \"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f\": rpc error: code = NotFound desc = could not find container \"41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f\": container with ID starting with 41cde784576d39fc12a97f369155c0ce6d8ccb1141d5275d7780557413968b1f not found: ID does not exist" Oct 31 01:23:25.373736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.373641 5011 scope.go:122] "RemoveContainer" containerID="2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf" Oct 31 01:23:25.384855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.384825 5011 scope.go:122] "RemoveContainer" containerID="2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf" Oct 31 01:23:25.385148 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:25.385101 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf\": container with ID starting with 2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf not found: ID does not exist" containerID="2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf" Oct 31 01:23:25.385285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.385141 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf"} err="failed to get container status \"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf\": rpc error: code = NotFound desc = could not find container \"2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf\": container with ID starting with 2e9ef7f4c89dc1742828c45f601cac52a7a30fd4daeedde1916cd7d306465caf not found: ID does not exist" Oct 31 01:23:25.404526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.404495 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:23:25.404671 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.404559 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:23:25.419865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.419829 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 01:23:25.426801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.426772 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173beed-90e9-49b3-82a8-6404a4d943c6-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.426801 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.426800 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13cb12cf-6050-4d2a-be4b-01a159d5a3c6-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.434554 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.434529 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-w3-daaf26-kserve-56f479c94-l7wb5"] Oct 31 01:23:25.473049 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.473007 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 01:23:25.479583 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.479558 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d"] Oct 31 01:23:25.516882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.516856 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 01:23:25.522066 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.522033 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/simulator-w3-daaf26-kserve-6d88b7f9c8-h4kp5"] Oct 31 01:23:25.671810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.671715 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h"] Oct 31 01:23:25.672248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672228 5011 cpu_manager.go:414] "containerMap: removing container" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" Oct 31 01:23:25.672248 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672244 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672260 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672265 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672273 5011 cpu_manager.go:414] "containerMap: removing container" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672279 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672288 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6743fcc6-aa1e-4a7b-89e0-3414e93e8309" containerName="revoke-keys" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672293 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6743fcc6-aa1e-4a7b-89e0-3414e93e8309" containerName="revoke-keys" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672322 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672327 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672390 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672400 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="74407c2a-2b5d-4185-955e-b47c4e73d766" containerName="istio-proxy" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672405 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6743fcc6-aa1e-4a7b-89e0-3414e93e8309" containerName="revoke-keys" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672416 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" Oct 31 01:23:25.672423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.672425 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" containerName="main" Oct 31 01:23:25.677564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.677540 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.685739 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.685712 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cb12cf-6050-4d2a-be4b-01a159d5a3c6" path="/var/lib/kubelet/pods/13cb12cf-6050-4d2a-be4b-01a159d5a3c6/volumes" Oct 31 01:23:25.686200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.686180 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6173beed-90e9-49b3-82a8-6404a4d943c6" path="/var/lib/kubelet/pods/6173beed-90e9-49b3-82a8-6404a4d943c6/volumes" Oct 31 01:23:25.686639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.686620 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" path="/var/lib/kubelet/pods/bdb915eb-5c6f-4f70-99a8-7a7bdac856a6/volumes" Oct 31 01:23:25.694601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.694572 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:23:25.694724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.694649 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:23:25.695147 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.695125 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h"] Oct 31 01:23:25.720234 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.720177 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:25.720375 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.720360 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:25.831897 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.831863 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.832223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.831933 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6dxw\" (UniqueName: \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.832223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.832065 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.933418 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.933335 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6dxw\" (UniqueName: \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.933574 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.933483 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.933625 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.933570 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.934279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.934248 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.936167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.936140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.957108 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.957076 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6dxw\" (UniqueName: \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:25.997908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:25.997884 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:26.064315 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:26.063650 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk"] Oct 31 01:23:26.072055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:26.072026 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fdcea1-8trwk"] Oct 31 01:23:26.105257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:26.105198 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/distinct-2-w3-daaf26-kserve-868bb44f9c-9dv6d" podUID="bdb915eb-5c6f-4f70-99a8-7a7bdac856a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:23:26.306694 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:26.306662 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h"] Oct 31 01:23:26.307898 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:26.307866 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90344cce_31dc_4965_97dd_da50eea42dec.slice/crio-14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9 WatchSource:0}: Error finding container 14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9: Status 404 returned error can't find the container with id 14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9 Oct 31 01:23:26.354887 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:26.354863 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" event={"ID":"90344cce-31dc-4965-97dd-da50eea42dec","Type":"ContainerStarted","Data":"14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9"} Oct 31 01:23:27.106854 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.106818 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd"] Oct 31 01:23:27.110834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.110810 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.132103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.132075 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd"] Oct 31 01:23:27.248018 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.247985 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.248156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.248121 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.248355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.248330 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shq7l\" (UniqueName: \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.349590 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.349557 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shq7l\" (UniqueName: \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.349730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.349652 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.349730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.349722 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.350431 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.350403 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.351944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.351925 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.366776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.366715 5011 generic.go:356] "Generic (PLEG): container finished" podID="90344cce-31dc-4965-97dd-da50eea42dec" containerID="c2992f62190f688f43830cd6c1c21d6d8fcab507945e6897bb2c1b1fad02d695" exitCode=0 Oct 31 01:23:27.366877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.366794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" event={"ID":"90344cce-31dc-4965-97dd-da50eea42dec","Type":"ContainerDied","Data":"c2992f62190f688f43830cd6c1c21d6d8fcab507945e6897bb2c1b1fad02d695"} Oct 31 01:23:27.373220 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.373181 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shq7l\" (UniqueName: \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\") pod \"maas-api-revoke-sub-41c5f78e-97ktd\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.431949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.431924 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:27.683091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.683015 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0424739b-f68f-4ebf-80f7-9f6cb7f621f2" path="/var/lib/kubelet/pods/0424739b-f68f-4ebf-80f7-9f6cb7f621f2/volumes" Oct 31 01:23:27.747653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:27.747628 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd"] Oct 31 01:23:27.749638 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:27.749602 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42eee6bc_a7a4_412d_9974_7e13fbbe49b5.slice/crio-ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f WatchSource:0}: Error finding container ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f: Status 404 returned error can't find the container with id ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f Oct 31 01:23:28.375380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.375338 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" event={"ID":"42eee6bc-a7a4-412d-9974-7e13fbbe49b5","Type":"ContainerStarted","Data":"265385f8010b84a5e63c0163f2ab8cac3f7fca9f9bedfe6295021c1f29b4cdaf"} Oct 31 01:23:28.375771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.375405 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" event={"ID":"42eee6bc-a7a4-412d-9974-7e13fbbe49b5","Type":"ContainerStarted","Data":"ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f"} Oct 31 01:23:28.419459 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.419391 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" podStartSLOduration=1.419369237 podStartE2EDuration="1.419369237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:28.41532379 +0000 UTC m=+3876.864411439" watchObservedRunningTime="2026-10-31 01:23:28.419369237 +0000 UTC m=+3876.868456880" Oct 31 01:23:28.522947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.522917 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:28.669003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.668974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\" (UniqueName: \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\") pod \"90344cce-31dc-4965-97dd-da50eea42dec\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " Oct 31 01:23:28.669140 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.669010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\") pod \"90344cce-31dc-4965-97dd-da50eea42dec\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " Oct 31 01:23:28.669140 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.669054 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\") pod \"90344cce-31dc-4965-97dd-da50eea42dec\" (UID: \"90344cce-31dc-4965-97dd-da50eea42dec\") " Oct 31 01:23:28.669526 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.669497 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca" pod "90344cce-31dc-4965-97dd-da50eea42dec" (UID: "90344cce-31dc-4965-97dd-da50eea42dec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:28.671091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.671068 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw" pod "90344cce-31dc-4965-97dd-da50eea42dec" (UID: "90344cce-31dc-4965-97dd-da50eea42dec"). InnerVolumeSpecName "kube-api-access-g6dxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:28.671174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.671071 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca" pod "90344cce-31dc-4965-97dd-da50eea42dec" (UID: "90344cce-31dc-4965-97dd-da50eea42dec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:28.770780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.770743 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6dxw\" (UniqueName: \"kubernetes.io/projected/90344cce-31dc-4965-97dd-da50eea42dec-kube-api-access-g6dxw\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:28.770780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.770778 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:28.770966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:28.770789 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90344cce-31dc-4965-97dd-da50eea42dec-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:29.383414 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:29.383379 5011 generic.go:356] "Generic (PLEG): container finished" podID="42eee6bc-a7a4-412d-9974-7e13fbbe49b5" containerID="265385f8010b84a5e63c0163f2ab8cac3f7fca9f9bedfe6295021c1f29b4cdaf" exitCode=0 Oct 31 01:23:29.383853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:29.383471 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" event={"ID":"42eee6bc-a7a4-412d-9974-7e13fbbe49b5","Type":"ContainerDied","Data":"265385f8010b84a5e63c0163f2ab8cac3f7fca9f9bedfe6295021c1f29b4cdaf"} Oct 31 01:23:29.384904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:29.384881 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" Oct 31 01:23:29.384904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:29.384887 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h" event={"ID":"90344cce-31dc-4965-97dd-da50eea42dec","Type":"ContainerDied","Data":"14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9"} Oct 31 01:23:29.385068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:29.384925 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f505105b3927a73211b1540c6fd2fcbba06ed3481bdfe62c88ad3e642699e9" Oct 31 01:23:30.285190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.285126 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:23:30.386036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386002 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386055 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386096 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386140 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386196 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386635 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386457 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:30.386702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386484 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:30.386702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386642 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:30.386894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386844 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.386894 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386876 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\") pod \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\" (UID: \"cb3019ac-6db7-4407-9e3b-4329b9de1df5\") " Oct 31 01:23:30.387075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.386996 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:30.387406 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.387370 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.387533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.387409 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.387533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.387422 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.387533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.387436 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.387533 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.387498 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:30.389916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.389440 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:30.389916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.389568 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "kube-api-access-g7ckz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:30.389916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.389639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:30.389916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.389728 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo" pod "cb3019ac-6db7-4407-9e3b-4329b9de1df5" (UID: "cb3019ac-6db7-4407-9e3b-4329b9de1df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:30.395268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.395245 5011 generic.go:356] "Generic (PLEG): container finished" podID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerID="ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec" exitCode=0 Oct 31 01:23:30.395355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.395338 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" Oct 31 01:23:30.395410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.395379 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" event={"ID":"cb3019ac-6db7-4407-9e3b-4329b9de1df5","Type":"ContainerDied","Data":"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec"} Oct 31 01:23:30.395461 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.395442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg" event={"ID":"cb3019ac-6db7-4407-9e3b-4329b9de1df5","Type":"ContainerDied","Data":"70593d22b10970d132c2e3149923afdbf6bbb070220f8ab2c4f9c4de5f5b409a"} Oct 31 01:23:30.395560 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.395474 5011 scope.go:122] "RemoveContainer" containerID="ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec" Oct 31 01:23:30.417506 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.417483 5011 scope.go:122] "RemoveContainer" containerID="ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec" Oct 31 01:23:30.417947 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:30.417922 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec\": container with ID starting with ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec not found: ID does not exist" containerID="ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec" Oct 31 01:23:30.418031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.417959 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec"} err="failed to get container status \"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec\": rpc error: code = NotFound desc = could not find container \"ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec\": container with ID starting with ec7f92a4eac18804cab12f742dad7e44574f362b1bbe9bb4c7b630f215c309ec not found: ID does not exist" Oct 31 01:23:30.450662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.450629 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:23:30.456383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.456354 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-mn4mg"] Oct 31 01:23:30.489285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.489186 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ckz\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-kube-api-access-g7ckz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.489285 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.489274 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.489485 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.489290 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.489485 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.489304 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3019ac-6db7-4407-9e3b-4329b9de1df5-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.489485 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.489319 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3019ac-6db7-4407-9e3b-4329b9de1df5-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.537330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.537270 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:30.691052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.691023 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\" (UniqueName: \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\") pod \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " Oct 31 01:23:30.691245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.691059 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\") pod \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " Oct 31 01:23:30.691245 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.691104 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\") pod \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\" (UID: \"42eee6bc-a7a4-412d-9974-7e13fbbe49b5\") " Oct 31 01:23:30.691547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.691517 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca" pod "42eee6bc-a7a4-412d-9974-7e13fbbe49b5" (UID: "42eee6bc-a7a4-412d-9974-7e13fbbe49b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:30.693225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.693181 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca" pod "42eee6bc-a7a4-412d-9974-7e13fbbe49b5" (UID: "42eee6bc-a7a4-412d-9974-7e13fbbe49b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:30.693338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.693318 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l" pod "42eee6bc-a7a4-412d-9974-7e13fbbe49b5" (UID: "42eee6bc-a7a4-412d-9974-7e13fbbe49b5"). InnerVolumeSpecName "kube-api-access-shq7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:30.792647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.792580 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shq7l\" (UniqueName: \"kubernetes.io/projected/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-kube-api-access-shq7l\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.792647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.792607 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.792647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.792617 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42eee6bc-a7a4-412d-9974-7e13fbbe49b5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.949976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:30.949939 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:23:31.404864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:31.404831 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" Oct 31 01:23:31.404864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:31.404847 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd" event={"ID":"42eee6bc-a7a4-412d-9974-7e13fbbe49b5","Type":"ContainerDied","Data":"ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f"} Oct 31 01:23:31.405474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:31.404891 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae619c3af1f318c053f9e1e8f2d3526f1c4daa4d4b9ada415b854accae28906f" Oct 31 01:23:31.683888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:31.683809 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" path="/var/lib/kubelet/pods/cb3019ac-6db7-4407-9e3b-4329b9de1df5/volumes" Oct 31 01:23:31.946268 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:31.946156 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:23:33.034421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:33.034386 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb"] Oct 31 01:23:33.046073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:33.046041 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0432b9-kzmcb"] Oct 31 01:23:33.683588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:33.683552 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d47532-dbb5-4b7e-8934-a96fd7043ae3" path="/var/lib/kubelet/pods/23d47532-dbb5-4b7e-8934-a96fd7043ae3/volumes" Oct 31 01:23:34.875255 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:34.867498 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5"] Oct 31 01:23:34.879309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:34.879278 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8f0029-964v5"] Oct 31 01:23:35.681874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:35.681841 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6743fcc6-aa1e-4a7b-89e0-3414e93e8309" path="/var/lib/kubelet/pods/6743fcc6-aa1e-4a7b-89e0-3414e93e8309/volumes" Oct 31 01:23:38.710384 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:38.710348 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29"] Oct 31 01:23:38.722444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:38.722411 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-daaf26-l7m29"] Oct 31 01:23:39.683704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:39.683667 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1a8807-acba-45d1-9e16-2c0090eb525c" path="/var/lib/kubelet/pods/fe1a8807-acba-45d1-9e16-2c0090eb525c/volumes" Oct 31 01:23:42.772941 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.772905 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:23:42.773458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.773166 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" containerID="cri-o://d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d" gracePeriod=30 Oct 31 01:23:42.786398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.785544 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:23:42.786398 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.785770 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" containerID="cri-o://e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820" gracePeriod=30 Oct 31 01:23:42.790129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.790100 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:42.790337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.790309 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" containerID="cri-o://3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff" gracePeriod=30 Oct 31 01:23:42.797178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.797151 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:23:42.797403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:42.797368 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" containerID="cri-o://d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57" gracePeriod=30 Oct 31 01:23:44.471218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:44.471171 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h"] Oct 31 01:23:44.482601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:44.482567 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-328d61-2v26h"] Oct 31 01:23:45.683709 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:45.683674 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90344cce-31dc-4965-97dd-da50eea42dec" path="/var/lib/kubelet/pods/90344cce-31dc-4965-97dd-da50eea42dec/volumes" Oct 31 01:23:46.083855 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:46.083821 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 01:23:46.084225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:46.084173 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" containerID="cri-o://ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848" gracePeriod=30 Oct 31 01:23:48.087615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.087580 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:23:48.183476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.183450 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:48.185482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.185450 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:23:48.187727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.187649 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:23:48.230769 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.230739 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.230909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.230806 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.230909 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.230889 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.230921 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.230998 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231011 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.231144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231041 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231075 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231144 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231125 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231146 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.231400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231216 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.231400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231231 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\") pod \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\" (UID: \"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04\") " Oct 31 01:23:48.231400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231388 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.231703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231593 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.231703 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231689 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.231844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231708 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.231844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231722 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.231844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231734 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.231844 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.231749 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.233580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.233521 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.233685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.233588 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.233873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.233849 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.234052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.234036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s" pod "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" (UID: "ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04"). InnerVolumeSpecName "kube-api-access-spl5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332002 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332059 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332156 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332245 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332299 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332326 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332366 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332398 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332430 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332473 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332564 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332592 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332618 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332646 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332677 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332712 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332744 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332775 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332820 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332855 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332887 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332909 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332944 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.332980 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\") pod \"842bb9ce-48b0-4766-aa72-1507a421e8ef\" (UID: \"842bb9ce-48b0-4766-aa72-1507a421e8ef\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333000 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\") pod \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\" (UID: \"47dd24a0-8686-42ed-9dd5-18adfe9073e3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333027 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\") pod \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\" (UID: \"d31bbad8-d7c6-4092-a7ee-e7484076a9a3\") " Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333361 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333381 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spl5s\" (UniqueName: \"kubernetes.io/projected/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-kube-api-access-spl5s\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333391 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.333757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.333401 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.335881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.335623 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.335881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.335844 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.336720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.336690 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.337093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.336739 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "kube-api-access-rtrrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.337093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.336963 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.337578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.337366 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.337706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.337682 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.338096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.337760 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.338096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.337956 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.338096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.337999 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.338540 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.338515 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.338802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.338780 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.339334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.338810 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.339631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.339608 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.339832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.339665 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.339999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.339968 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.339999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.338813 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.339999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.338930 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.339999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.339277 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.341699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.341669 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.341981 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.341958 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.342074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.341943 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.342288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.342262 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.342523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.342499 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "kube-api-access-c8hw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.343241 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.343188 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy" pod "d31bbad8-d7c6-4092-a7ee-e7484076a9a3" (UID: "d31bbad8-d7c6-4092-a7ee-e7484076a9a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.343721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.343697 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6" pod "842bb9ce-48b0-4766-aa72-1507a421e8ef" (UID: "842bb9ce-48b0-4766-aa72-1507a421e8ef"). InnerVolumeSpecName "kube-api-access-56wj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.343908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.343890 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo" pod "47dd24a0-8686-42ed-9dd5-18adfe9073e3" (UID: "47dd24a0-8686-42ed-9dd5-18adfe9073e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.434523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434487 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434517 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434529 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434539 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434552 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434565 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434582 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434594 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434610 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434624 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8hw2\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-kube-api-access-c8hw2\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434637 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434655 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434668 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434681 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434695 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842bb9ce-48b0-4766-aa72-1507a421e8ef-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434706 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434719 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrrf\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-kube-api-access-rtrrf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434733 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434746 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434757 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434765 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31bbad8-d7c6-4092-a7ee-e7484076a9a3-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434773 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434780 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434793 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47dd24a0-8686-42ed-9dd5-18adfe9073e3-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434807 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56wj6\" (UniqueName: \"kubernetes.io/projected/842bb9ce-48b0-4766-aa72-1507a421e8ef-kube-api-access-56wj6\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434820 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.434828 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.434835 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842bb9ce-48b0-4766-aa72-1507a421e8ef-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.560520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.560476 5011 generic.go:356] "Generic (PLEG): container finished" podID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerID="3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff" exitCode=0 Oct 31 01:23:48.560690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.560522 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" Oct 31 01:23:48.560690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.560556 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" event={"ID":"47dd24a0-8686-42ed-9dd5-18adfe9073e3","Type":"ContainerDied","Data":"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff"} Oct 31 01:23:48.560690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.560611 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7" event={"ID":"47dd24a0-8686-42ed-9dd5-18adfe9073e3","Type":"ContainerDied","Data":"fd6e2ab0528d6b44fea2cf65b6c30dcf93b5df55be32ec6531abe8f6baa649c6"} Oct 31 01:23:48.560690 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.560640 5011 scope.go:122] "RemoveContainer" containerID="3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff" Oct 31 01:23:48.562104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.562078 5011 generic.go:356] "Generic (PLEG): container finished" podID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerID="d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57" exitCode=0 Oct 31 01:23:48.562279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.562113 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" Oct 31 01:23:48.562279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.562151 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" event={"ID":"842bb9ce-48b0-4766-aa72-1507a421e8ef","Type":"ContainerDied","Data":"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57"} Oct 31 01:23:48.562279 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.562202 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9" event={"ID":"842bb9ce-48b0-4766-aa72-1507a421e8ef","Type":"ContainerDied","Data":"59782096e5bf7e651eb38ab06bf41010c3bc9c6f84a15965f35656deb7e557dc"} Oct 31 01:23:48.563907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.563883 5011 generic.go:356] "Generic (PLEG): container finished" podID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerID="d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d" exitCode=0 Oct 31 01:23:48.564046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.563926 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" Oct 31 01:23:48.564046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.563981 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" event={"ID":"d31bbad8-d7c6-4092-a7ee-e7484076a9a3","Type":"ContainerDied","Data":"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d"} Oct 31 01:23:48.564046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.564018 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg" event={"ID":"d31bbad8-d7c6-4092-a7ee-e7484076a9a3","Type":"ContainerDied","Data":"dac1dffa7b00c53462b6dea69aaeb27313121314449629e8b1b68160b35e4e8b"} Oct 31 01:23:48.566818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.566793 5011 generic.go:356] "Generic (PLEG): container finished" podID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerID="e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820" exitCode=0 Oct 31 01:23:48.566959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.566835 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" Oct 31 01:23:48.566959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.566868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" event={"ID":"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04","Type":"ContainerDied","Data":"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820"} Oct 31 01:23:48.566959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.566904 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg" event={"ID":"ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04","Type":"ContainerDied","Data":"bae8604185e639057a29763b16437d2c890840dd48e170be1c275d9ab45dad75"} Oct 31 01:23:48.582287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.582270 5011 scope.go:122] "RemoveContainer" containerID="3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff" Oct 31 01:23:48.582621 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:48.582602 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff\": container with ID starting with 3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff not found: ID does not exist" containerID="3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff" Oct 31 01:23:48.582721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.582631 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff"} err="failed to get container status \"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff\": rpc error: code = NotFound desc = could not find container \"3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff\": container with ID starting with 3cb6a46536eb9a631170a8a69013068736eee6929515cd1ec8fe1ae6bdcb46ff not found: ID does not exist" Oct 31 01:23:48.582721 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.582653 5011 scope.go:122] "RemoveContainer" containerID="d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57" Oct 31 01:23:48.594277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.594182 5011 scope.go:122] "RemoveContainer" containerID="d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57" Oct 31 01:23:48.594490 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:48.594473 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57\": container with ID starting with d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57 not found: ID does not exist" containerID="d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57" Oct 31 01:23:48.594566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.594501 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57"} err="failed to get container status \"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57\": rpc error: code = NotFound desc = could not find container \"d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57\": container with ID starting with d53c84c0319b49ba03d401997cc7c2c7897809c3b494a06b47ffde86e9553f57 not found: ID does not exist" Oct 31 01:23:48.594566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.594522 5011 scope.go:122] "RemoveContainer" containerID="d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d" Oct 31 01:23:48.603886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.603870 5011 scope.go:122] "RemoveContainer" containerID="d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d" Oct 31 01:23:48.604165 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:48.604139 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d\": container with ID starting with d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d not found: ID does not exist" containerID="d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d" Oct 31 01:23:48.604260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.604171 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d"} err="failed to get container status \"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d\": rpc error: code = NotFound desc = could not find container \"d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d\": container with ID starting with d60e2fb6f5eee30dc26d066296dd9849e1e6f919c8ccefad6fa9afd250aa4f8d not found: ID does not exist" Oct 31 01:23:48.604260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.604187 5011 scope.go:122] "RemoveContainer" containerID="e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820" Oct 31 01:23:48.614217 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.614191 5011 scope.go:122] "RemoveContainer" containerID="e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820" Oct 31 01:23:48.614503 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:48.614485 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820\": container with ID starting with e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820 not found: ID does not exist" containerID="e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820" Oct 31 01:23:48.614586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.614507 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820"} err="failed to get container status \"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820\": rpc error: code = NotFound desc = could not find container \"e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820\": container with ID starting with e1bdfc9101723c7946a30109fb5d42f7e8bb6ceee02fa17e497dec0e6cfa1820 not found: ID does not exist" Oct 31 01:23:48.662058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.662030 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:48.675332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.675306 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-mstv7"] Oct 31 01:23:48.709633 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.709605 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:23:48.713249 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.713221 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-6gfxg"] Oct 31 01:23:48.749416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.749351 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:23:48.752604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.752584 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-s5vzg"] Oct 31 01:23:48.787396 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.787373 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:23:48.791449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:48.791418 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-9j9s9"] Oct 31 01:23:49.682674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:49.682642 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" path="/var/lib/kubelet/pods/47dd24a0-8686-42ed-9dd5-18adfe9073e3/volumes" Oct 31 01:23:49.683127 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:49.683113 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" path="/var/lib/kubelet/pods/842bb9ce-48b0-4766-aa72-1507a421e8ef/volumes" Oct 31 01:23:49.683555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:49.683540 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" path="/var/lib/kubelet/pods/d31bbad8-d7c6-4092-a7ee-e7484076a9a3/volumes" Oct 31 01:23:49.684258 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:49.684244 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" path="/var/lib/kubelet/pods/ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04/volumes" Oct 31 01:23:51.360752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.360677 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 01:23:51.465462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465425 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465476 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465503 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465578 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465630 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465684 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465712 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465744 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465783 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\") pod \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\" (UID: \"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1\") " Oct 31 01:23:51.465899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465871 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:51.466178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465890 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:51.466178 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.465992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:51.466289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.466227 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.466289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.466251 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.466289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.466265 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.466609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.466290 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:51.467655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.467612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:51.468156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.468129 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:51.468293 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.468253 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:51.468382 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.468361 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "kube-api-access-grzkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:51.468640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.468620 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy" pod "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" (UID: "e97d2c5b-f201-4aa7-bccb-e03b6cf994d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:51.567100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567071 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.567100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567097 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.567100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567108 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.567363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567116 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.567363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567124 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.567363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.567132 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzkp\" (UniqueName: \"kubernetes.io/projected/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1-kube-api-access-grzkp\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:51.597235 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.597189 5011 generic.go:356] "Generic (PLEG): container finished" podID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerID="ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848" exitCode=0 Oct 31 01:23:51.597372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.597262 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" Oct 31 01:23:51.597372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.597272 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" event={"ID":"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1","Type":"ContainerDied","Data":"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848"} Oct 31 01:23:51.597372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.597324 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb" event={"ID":"e97d2c5b-f201-4aa7-bccb-e03b6cf994d1","Type":"ContainerDied","Data":"7ccfa90a0ca49b913dd4e212ffec5477b175a1f4aefdc2f600bd4793b0e6e5c2"} Oct 31 01:23:51.597372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.597346 5011 scope.go:122] "RemoveContainer" containerID="ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848" Oct 31 01:23:51.613717 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.613687 5011 scope.go:122] "RemoveContainer" containerID="ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848" Oct 31 01:23:51.614062 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:23:51.614040 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848\": container with ID starting with ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848 not found: ID does not exist" containerID="ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848" Oct 31 01:23:51.614125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.614072 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848"} err="failed to get container status \"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848\": rpc error: code = NotFound desc = could not find container \"ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848\": container with ID starting with ed5d5b5bd20db564807a98adad247d90b01cf67033d24a0ca1696ef04c411848 not found: ID does not exist" Oct 31 01:23:51.710116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.710081 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 01:23:51.749940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:51.749911 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-h8kwb"] Oct 31 01:23:52.519068 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:52.518915 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:23:52.519918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:52.519887 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" containerID="cri-o://06d7c76a75cc9546c064ab10196acede9916adf8124041b397660ea94408fa6a" gracePeriod=30 Oct 31 01:23:52.592995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:52.592960 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:23:52.735102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:52.735070 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:23:52.758008 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:52.757985 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:23:53.305455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305418 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp"] Oct 31 01:23:53.305934 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305911 5011 cpu_manager.go:414] "containerMap: removing container" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305935 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305957 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305965 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305985 5011 cpu_manager.go:414] "containerMap: removing container" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.305993 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306010 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306019 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306032 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" Oct 31 01:23:53.306043 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306040 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306056 5011 cpu_manager.go:414] "containerMap: removing container" podUID="42eee6bc-a7a4-412d-9974-7e13fbbe49b5" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306064 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42eee6bc-a7a4-412d-9974-7e13fbbe49b5" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306081 5011 cpu_manager.go:414] "containerMap: removing container" podUID="90344cce-31dc-4965-97dd-da50eea42dec" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306089 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90344cce-31dc-4965-97dd-da50eea42dec" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306105 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306112 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306243 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3019ac-6db7-4407-9e3b-4329b9de1df5" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306259 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="842bb9ce-48b0-4766-aa72-1507a421e8ef" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306269 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306279 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31bbad8-d7c6-4092-a7ee-e7484076a9a3" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306291 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="42eee6bc-a7a4-412d-9974-7e13fbbe49b5" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306302 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="90344cce-31dc-4965-97dd-da50eea42dec" containerName="revoke-keys" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306312 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4828e3-763a-4fd5-bd5d-74b3c5bb8c04" containerName="istio-proxy" Oct 31 01:23:53.306582 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.306325 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dd24a0-8686-42ed-9dd5-18adfe9073e3" containerName="istio-proxy" Oct 31 01:23:53.311264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.311241 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.332225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.328998 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:53.332225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.329137 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:23:53.332702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.332672 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp"] Oct 31 01:23:53.339670 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.339642 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:53.383913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.383879 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.384089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.383918 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.384089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.383937 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f5b\" (UniqueName: \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.432480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.432452 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:23:53.485139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.485096 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.485139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.485134 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f5b\" (UniqueName: \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.485363 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.485295 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.485853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.485827 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.487763 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.487740 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.509606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.509575 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f5b\" (UniqueName: \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.625932 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.625904 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:53.690426 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.689442 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97d2c5b-f201-4aa7-bccb-e03b6cf994d1" path="/var/lib/kubelet/pods/e97d2c5b-f201-4aa7-bccb-e03b6cf994d1/volumes" Oct 31 01:23:53.935035 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:53.935001 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp"] Oct 31 01:23:53.936327 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:23:53.936299 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607d7428_471e_4e63_b289_4246090e64fd.slice/crio-154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b WatchSource:0}: Error finding container 154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b: Status 404 returned error can't find the container with id 154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b Oct 31 01:23:54.625360 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:54.625318 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" event={"ID":"607d7428-471e-4e63-b289-4246090e64fd","Type":"ContainerStarted","Data":"4fff0c033ede42c04d604376465050bd574571ed7ede27d8d3a22fd978c91b04"} Oct 31 01:23:54.625562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:54.625379 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" event={"ID":"607d7428-471e-4e63-b289-4246090e64fd","Type":"ContainerStarted","Data":"154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b"} Oct 31 01:23:54.666538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:54.666490 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" podStartSLOduration=1.66647789 podStartE2EDuration="1.66647789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:54.665758058 +0000 UTC m=+3903.114845698" watchObservedRunningTime="2026-10-31 01:23:54.66647789 +0000 UTC m=+3903.115565566" Oct 31 01:23:55.380476 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.380442 5011 scope.go:122] "RemoveContainer" containerID="041b4d3eed368e92f9ed318870d113f3e0f99299c91cb62c541db0988286a911" Oct 31 01:23:55.392601 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.392557 5011 scope.go:122] "RemoveContainer" containerID="39634968511c2996269feaab63b5e02c6dde77a37e24be5a24979bf4a306abce" Oct 31 01:23:55.574623 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.574586 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:23:55.574824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.574657 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:23:55.634815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.634737 5011 generic.go:356] "Generic (PLEG): container finished" podID="607d7428-471e-4e63-b289-4246090e64fd" containerID="4fff0c033ede42c04d604376465050bd574571ed7ede27d8d3a22fd978c91b04" exitCode=0 Oct 31 01:23:55.634815 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:55.634776 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" event={"ID":"607d7428-471e-4e63-b289-4246090e64fd","Type":"ContainerDied","Data":"4fff0c033ede42c04d604376465050bd574571ed7ede27d8d3a22fd978c91b04"} Oct 31 01:23:56.780031 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.780006 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:56.824896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.824867 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\" (UniqueName: \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\") pod \"607d7428-471e-4e63-b289-4246090e64fd\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " Oct 31 01:23:56.825041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.824955 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\") pod \"607d7428-471e-4e63-b289-4246090e64fd\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " Oct 31 01:23:56.825117 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.825077 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\") pod \"607d7428-471e-4e63-b289-4246090e64fd\" (UID: \"607d7428-471e-4e63-b289-4246090e64fd\") " Oct 31 01:23:56.825349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.825327 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca" pod "607d7428-471e-4e63-b289-4246090e64fd" (UID: "607d7428-471e-4e63-b289-4246090e64fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:56.827308 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.827232 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca" pod "607d7428-471e-4e63-b289-4246090e64fd" (UID: "607d7428-471e-4e63-b289-4246090e64fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:56.827578 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.827553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b" pod "607d7428-471e-4e63-b289-4246090e64fd" (UID: "607d7428-471e-4e63-b289-4246090e64fd"). InnerVolumeSpecName "kube-api-access-44f5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:56.926488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.926421 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f5b\" (UniqueName: \"kubernetes.io/projected/607d7428-471e-4e63-b289-4246090e64fd-kube-api-access-44f5b\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:56.926488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.926448 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607d7428-471e-4e63-b289-4246090e64fd-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:56.926488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:56.926460 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607d7428-471e-4e63-b289-4246090e64fd-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.654359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.654317 5011 generic.go:356] "Generic (PLEG): container finished" podID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerID="06d7c76a75cc9546c064ab10196acede9916adf8124041b397660ea94408fa6a" exitCode=0 Oct 31 01:23:57.654519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.654390 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerDied","Data":"06d7c76a75cc9546c064ab10196acede9916adf8124041b397660ea94408fa6a"} Oct 31 01:23:57.654519 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.654470 5011 scope.go:122] "RemoveContainer" containerID="ab09e36446414bedbe1f0b4568de590f69c1ae8f032a69e09ec16ac8d3f6719c" Oct 31 01:23:57.656738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.656709 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" event={"ID":"607d7428-471e-4e63-b289-4246090e64fd","Type":"ContainerDied","Data":"154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b"} Oct 31 01:23:57.656873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.656751 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp" Oct 31 01:23:57.656873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.656758 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154f9f53af122afcf9946837b2ca0be544affc083dac107ab92792f5f9f5a00b" Oct 31 01:23:57.807115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.807090 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:23:57.840083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840057 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840097 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840129 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840176 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840230 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840282 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840435 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840387 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840427 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:57.840663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840438 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:57.840663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840435 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\") pod \"1c611828-648f-4f9a-968c-1da2c3b03d84\" (UID: \"1c611828-648f-4f9a-968c-1da2c3b03d84\") " Oct 31 01:23:57.840663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840505 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:57.840884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840748 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:57.840884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840768 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:57.840963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840934 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.840963 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840956 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.841048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840970 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.841048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.840985 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.841048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.841001 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c611828-648f-4f9a-968c-1da2c3b03d84-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.842439 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.842417 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:57.842549 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.842455 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:57.842741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.842725 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "kube-api-access-qjwcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:57.842813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.842791 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token" pod "1c611828-648f-4f9a-968c-1da2c3b03d84" (UID: "1c611828-648f-4f9a-968c-1da2c3b03d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:57.941752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.941720 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.941752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.941750 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwcn\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-kube-api-access-qjwcn\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.941947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.941763 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:57.941947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:57.941772 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c611828-648f-4f9a-968c-1da2c3b03d84-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:23:58.064481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.064396 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 01:23:58.667501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.667473 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" Oct 31 01:23:58.667501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.667489 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w" event={"ID":"1c611828-648f-4f9a-968c-1da2c3b03d84","Type":"ContainerDied","Data":"326c0e6bc8d5406c0146e2bebd13522ce995da9bed0413349fe140acf8b2f54a"} Oct 31 01:23:58.667746 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.667590 5011 scope.go:122] "RemoveContainer" containerID="06d7c76a75cc9546c064ab10196acede9916adf8124041b397660ea94408fa6a" Oct 31 01:23:58.720297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.720270 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:23:58.732111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:58.732076 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-sml6w"] Oct 31 01:23:59.681314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:23:59.681283 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" path="/var/lib/kubelet/pods/1c611828-648f-4f9a-968c-1da2c3b03d84/volumes" Oct 31 01:24:01.947976 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:01.947941 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:24:03.033416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.032935 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:24:03.033416 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.033176 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" containerID="cri-o://de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0" gracePeriod=30 Oct 31 01:24:03.689347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689312 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv"] Oct 31 01:24:03.689733 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689717 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:03.689798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689734 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:03.689798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689763 5011 cpu_manager.go:414] "containerMap: removing container" podUID="607d7428-471e-4e63-b289-4246090e64fd" containerName="revoke-keys" Oct 31 01:24:03.689798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689768 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607d7428-471e-4e63-b289-4246090e64fd" containerName="revoke-keys" Oct 31 01:24:03.689960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689829 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="607d7428-471e-4e63-b289-4246090e64fd" containerName="revoke-keys" Oct 31 01:24:03.689960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689838 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:03.689960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.689848 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:03.694785 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.694763 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.716244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.716200 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv"] Oct 31 01:24:03.727057 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.727029 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:03.727199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.727077 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:03.727199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.727110 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:03.727199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.727167 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:24:03.798639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.798600 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.798820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.798659 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.798820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.798746 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8v7\" (UniqueName: \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.900274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.900239 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.900274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.900275 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8v7\" (UniqueName: \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.900518 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.900384 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.900908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.900883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.902724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.902699 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:03.925655 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:03.925622 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8v7\" (UniqueName: \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:04.009881 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:04.009806 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:04.336133 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:24:04.336093 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b938ac_d329_4abb_8c3b_5f85d67ed22c.slice/crio-cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d WatchSource:0}: Error finding container cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d: Status 404 returned error can't find the container with id cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d Oct 31 01:24:04.336837 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:04.336806 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv"] Oct 31 01:24:04.713900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:04.713862 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" event={"ID":"53b938ac-d329-4abb-8c3b-5f85d67ed22c","Type":"ContainerStarted","Data":"0413fe374ff88169023040b8b5ba28c9966eb4afdc198094da2c32af2b7be1f4"} Oct 31 01:24:04.714074 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:04.713917 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" event={"ID":"53b938ac-d329-4abb-8c3b-5f85d67ed22c","Type":"ContainerStarted","Data":"cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d"} Oct 31 01:24:04.757804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:04.757760 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" podStartSLOduration=1.75774805 podStartE2EDuration="1.75774805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:04.755353524 +0000 UTC m=+3913.204441163" watchObservedRunningTime="2026-10-31 01:24:04.75774805 +0000 UTC m=+3913.206835688" Oct 31 01:24:05.722988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:05.722949 5011 generic.go:356] "Generic (PLEG): container finished" podID="53b938ac-d329-4abb-8c3b-5f85d67ed22c" containerID="0413fe374ff88169023040b8b5ba28c9966eb4afdc198094da2c32af2b7be1f4" exitCode=0 Oct 31 01:24:05.723527 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:05.723034 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" event={"ID":"53b938ac-d329-4abb-8c3b-5f85d67ed22c","Type":"ContainerDied","Data":"0413fe374ff88169023040b8b5ba28c9966eb4afdc198094da2c32af2b7be1f4"} Oct 31 01:24:06.051579 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.051494 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf"] Oct 31 01:24:06.051947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.051930 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:06.052022 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.051947 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c611828-648f-4f9a-968c-1da2c3b03d84" containerName="istio-proxy" Oct 31 01:24:06.055286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.055262 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.071768 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.071740 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf"] Oct 31 01:24:06.125668 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.125640 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.125835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.125756 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grv87\" (UniqueName: \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.125835 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.125810 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.227410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.227374 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grv87\" (UniqueName: \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.227553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.227435 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.227553 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.227537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.228110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.228072 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.230082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.230058 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.251580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.251550 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grv87\" (UniqueName: \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\") pod \"maas-api-revoke-sub-48ca66de-vr8qf\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.369584 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.369554 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:06.466111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.466080 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:24:06.466304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.466144 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:24:06.730696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.730664 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf"] Oct 31 01:24:06.861115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.861080 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:06.934157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.934125 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\") pod \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " Oct 31 01:24:06.934332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.934190 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\" (UniqueName: \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\") pod \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " Oct 31 01:24:06.934332 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.934267 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\") pod \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\" (UID: \"53b938ac-d329-4abb-8c3b-5f85d67ed22c\") " Oct 31 01:24:06.934916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.934882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca" pod "53b938ac-d329-4abb-8c3b-5f85d67ed22c" (UID: "53b938ac-d329-4abb-8c3b-5f85d67ed22c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:06.936737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.936703 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7" pod "53b938ac-d329-4abb-8c3b-5f85d67ed22c" (UID: "53b938ac-d329-4abb-8c3b-5f85d67ed22c"). InnerVolumeSpecName "kube-api-access-hq8v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:06.937103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:06.937079 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca" pod "53b938ac-d329-4abb-8c3b-5f85d67ed22c" (UID: "53b938ac-d329-4abb-8c3b-5f85d67ed22c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:07.035481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.035448 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:07.035606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.035483 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8v7\" (UniqueName: \"kubernetes.io/projected/53b938ac-d329-4abb-8c3b-5f85d67ed22c-kube-api-access-hq8v7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:07.035606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.035497 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b938ac-d329-4abb-8c3b-5f85d67ed22c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:07.473301 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.473267 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:24:07.740943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.740851 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" Oct 31 01:24:07.740943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.740881 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv" event={"ID":"53b938ac-d329-4abb-8c3b-5f85d67ed22c","Type":"ContainerDied","Data":"cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d"} Oct 31 01:24:07.740943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.740933 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7af0ca0863dd821960e7e953a473386fe51c763ed4d84db13686d453bcd41d" Oct 31 01:24:07.742878 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.742841 5011 generic.go:356] "Generic (PLEG): container finished" podID="aa2b21f6-1c79-4125-a034-2e86206a4fee" containerID="c30950f24af87f1407ff0fb9a0be2230a3a0cc997e6056cbc1100ba46b6da1dc" exitCode=0 Oct 31 01:24:07.743693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.743669 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" event={"ID":"aa2b21f6-1c79-4125-a034-2e86206a4fee","Type":"ContainerDied","Data":"c30950f24af87f1407ff0fb9a0be2230a3a0cc997e6056cbc1100ba46b6da1dc"} Oct 31 01:24:07.743794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:07.743714 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" event={"ID":"aa2b21f6-1c79-4125-a034-2e86206a4fee","Type":"ContainerStarted","Data":"c0f7393614705957ecb0d730d046590436bcd29e2dbef6669e35f15849e39884"} Oct 31 01:24:08.331604 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.331575 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:24:08.449287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449251 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449305 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449371 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449407 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449497 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449508 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449534 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449615 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449643 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.449730 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449676 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\") pod \"b74b1d56-7aca-41ab-812f-2dcc158a8134\" (UID: \"b74b1d56-7aca-41ab-812f-2dcc158a8134\") " Oct 31 01:24:08.450059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449873 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.450059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.449887 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.450168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.450076 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.450168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.450096 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.450168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.450110 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.450168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.450123 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.450486 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.450463 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:08.452394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.452363 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "kube-api-access-pg69s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:08.452506 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.452489 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:08.452588 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.452570 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.452659 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.452628 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo" pod "b74b1d56-7aca-41ab-812f-2dcc158a8134" (UID: "b74b1d56-7aca-41ab-812f-2dcc158a8134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:24:08.551010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.550973 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b1d56-7aca-41ab-812f-2dcc158a8134-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.551010 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.551007 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.551176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.551023 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.551176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.551035 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.551176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.551049 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg69s\" (UniqueName: \"kubernetes.io/projected/b74b1d56-7aca-41ab-812f-2dcc158a8134-kube-api-access-pg69s\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.752597 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.752556 5011 generic.go:356] "Generic (PLEG): container finished" podID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerID="de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0" exitCode=0 Oct 31 01:24:08.753082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.752589 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" event={"ID":"b74b1d56-7aca-41ab-812f-2dcc158a8134","Type":"ContainerDied","Data":"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0"} Oct 31 01:24:08.753082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.752617 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" Oct 31 01:24:08.753082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.752659 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx" event={"ID":"b74b1d56-7aca-41ab-812f-2dcc158a8134","Type":"ContainerDied","Data":"3a4468e4548001966605a283bc1557e2ec23c69769b7cc0e73c948e50b55e2ad"} Oct 31 01:24:08.753082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.752690 5011 scope.go:122] "RemoveContainer" containerID="de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0" Oct 31 01:24:08.766330 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.766200 5011 scope.go:122] "RemoveContainer" containerID="de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0" Oct 31 01:24:08.766613 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:08.766591 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0\": container with ID starting with de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0 not found: ID does not exist" containerID="de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0" Oct 31 01:24:08.766720 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.766621 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0"} err="failed to get container status \"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0\": rpc error: code = NotFound desc = could not find container \"de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0\": container with ID starting with de3e7dcace36b236002c936867cc2579da4f6f086ef7d39468bc3702a223c0a0 not found: ID does not exist" Oct 31 01:24:08.805446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.805416 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:24:08.819046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.819018 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-l8fgx"] Oct 31 01:24:08.895678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.895652 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:08.954496 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.954467 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\") pod \"aa2b21f6-1c79-4125-a034-2e86206a4fee\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " Oct 31 01:24:08.954676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.954557 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\" (UniqueName: \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\") pod \"aa2b21f6-1c79-4125-a034-2e86206a4fee\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " Oct 31 01:24:08.954676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.954612 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\") pod \"aa2b21f6-1c79-4125-a034-2e86206a4fee\" (UID: \"aa2b21f6-1c79-4125-a034-2e86206a4fee\") " Oct 31 01:24:08.955036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.955004 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca" pod "aa2b21f6-1c79-4125-a034-2e86206a4fee" (UID: "aa2b21f6-1c79-4125-a034-2e86206a4fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:08.956736 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.956704 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca" pod "aa2b21f6-1c79-4125-a034-2e86206a4fee" (UID: "aa2b21f6-1c79-4125-a034-2e86206a4fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:08.957357 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:08.957321 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87" pod "aa2b21f6-1c79-4125-a034-2e86206a4fee" (UID: "aa2b21f6-1c79-4125-a034-2e86206a4fee"). InnerVolumeSpecName "kube-api-access-grv87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:09.056130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.056040 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.056130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.056080 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grv87\" (UniqueName: \"kubernetes.io/projected/aa2b21f6-1c79-4125-a034-2e86206a4fee-kube-api-access-grv87\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.056130 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.056091 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2b21f6-1c79-4125-a034-2e86206a4fee-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.681699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.681665 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" path="/var/lib/kubelet/pods/b74b1d56-7aca-41ab-812f-2dcc158a8134/volumes" Oct 31 01:24:09.764176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.764146 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" Oct 31 01:24:09.764678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.764190 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf" event={"ID":"aa2b21f6-1c79-4125-a034-2e86206a4fee","Type":"ContainerDied","Data":"c0f7393614705957ecb0d730d046590436bcd29e2dbef6669e35f15849e39884"} Oct 31 01:24:09.764678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:09.764268 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f7393614705957ecb0d730d046590436bcd29e2dbef6669e35f15849e39884" Oct 31 01:24:13.085152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.085111 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" containerID="cri-o://b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971" gracePeriod=45 Oct 31 01:24:13.161193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.161157 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused" Oct 31 01:24:13.350677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.350600 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 01:24:13.503443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503413 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\" (UniqueName: \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503505 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503568 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\" (UniqueName: \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503597 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503688 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\") pod \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\" (UID: \"69adc0a6-30aa-4c5a-a120-656f7d569ca8\") " Oct 31 01:24:13.503865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503776 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:13.503865 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.503796 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:13.504037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.504009 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:13.504103 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.504077 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.504199 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.504107 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.506050 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.506022 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:13.506287 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.506267 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:13.506366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.506343 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl" pod "69adc0a6-30aa-4c5a-a120-656f7d569ca8" (UID: "69adc0a6-30aa-4c5a-a120-656f7d569ca8"). InnerVolumeSpecName "kube-api-access-l6vpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:13.604980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.604900 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69adc0a6-30aa-4c5a-a120-656f7d569ca8-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.604980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.604931 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.604980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.604939 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69adc0a6-30aa-4c5a-a120-656f7d569ca8-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.604980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.604948 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vpl\" (UniqueName: \"kubernetes.io/projected/69adc0a6-30aa-4c5a-a120-656f7d569ca8-kube-api-access-l6vpl\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:13.793819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.793780 5011 generic.go:356] "Generic (PLEG): container finished" podID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerID="b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971" exitCode=0 Oct 31 01:24:13.793986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.793832 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" Oct 31 01:24:13.793986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.793856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" event={"ID":"69adc0a6-30aa-4c5a-a120-656f7d569ca8","Type":"ContainerDied","Data":"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971"} Oct 31 01:24:13.793986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.793897 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn" event={"ID":"69adc0a6-30aa-4c5a-a120-656f7d569ca8","Type":"ContainerDied","Data":"b813f640797c4a5116fac0d7bd7e7ffd4031bb988e2a21a514f3ca54672ecf27"} Oct 31 01:24:13.793986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.793915 5011 scope.go:122] "RemoveContainer" containerID="b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971" Oct 31 01:24:13.803829 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.803812 5011 scope.go:122] "RemoveContainer" containerID="b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971" Oct 31 01:24:13.804094 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:13.804071 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971\": container with ID starting with b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971 not found: ID does not exist" containerID="b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971" Oct 31 01:24:13.804162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.804106 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971"} err="failed to get container status \"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971\": rpc error: code = NotFound desc = could not find container \"b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971\": container with ID starting with b3d93587ad5514954801316b708bd581f2687b9095ed09eb815534765da8d971 not found: ID does not exist" Oct 31 01:24:13.844814 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.844789 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 01:24:13.856530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:13.856471 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9e377/simulator-w5-c9e377-kserve-86bb57c78-tdpkn"] Oct 31 01:24:15.662602 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:15.662557 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv"] Oct 31 01:24:15.674073 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:15.674045 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2bdf3-894tv"] Oct 31 01:24:15.682289 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:15.682259 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b938ac-d329-4abb-8c3b-5f85d67ed22c" path="/var/lib/kubelet/pods/53b938ac-d329-4abb-8c3b-5f85d67ed22c/volumes" Oct 31 01:24:15.682820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:15.682803 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" path="/var/lib/kubelet/pods/69adc0a6-30aa-4c5a-a120-656f7d569ca8/volumes" Oct 31 01:24:19.153565 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:19.153522 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp"] Oct 31 01:24:19.165098 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:19.165061 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9e377-lmhpp"] Oct 31 01:24:19.682664 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:19.682627 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607d7428-471e-4e63-b289-4246090e64fd" path="/var/lib/kubelet/pods/607d7428-471e-4e63-b289-4246090e64fd/volumes" Oct 31 01:24:23.105971 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:23.105931 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:24:26.528884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:26.528848 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 01:24:26.529318 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:26.529114 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" containerID="cri-o://91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70" gracePeriod=30 Oct 31 01:24:29.206047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:29.206008 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 01:24:29.218254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:29.218225 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 01:24:31.787195 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.787166 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 01:24:31.876512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876472 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876512 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876517 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876539 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876582 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876633 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876697 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.876770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876762 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.877065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876791 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.877065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.876826 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\") pod \"b93750db-7a67-4128-8051-6aa3aa25254c\" (UID: \"b93750db-7a67-4128-8051-6aa3aa25254c\") " Oct 31 01:24:31.877223 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.877169 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:31.877386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.877356 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:31.877386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.877370 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:31.877624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.877521 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:31.877624 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.877576 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:31.879741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.879709 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:24:31.879825 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.879779 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:31.879964 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.879938 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:31.880027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.879984 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf" pod "b93750db-7a67-4128-8051-6aa3aa25254c" (UID: "b93750db-7a67-4128-8051-6aa3aa25254c"). InnerVolumeSpecName "kube-api-access-mllzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:31.932200 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.932164 5011 generic.go:356] "Generic (PLEG): container finished" podID="b93750db-7a67-4128-8051-6aa3aa25254c" containerID="91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70" exitCode=0 Oct 31 01:24:31.932355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.932299 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" Oct 31 01:24:31.932355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.932324 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" event={"ID":"b93750db-7a67-4128-8051-6aa3aa25254c","Type":"ContainerDied","Data":"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70"} Oct 31 01:24:31.932436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.932396 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl" event={"ID":"b93750db-7a67-4128-8051-6aa3aa25254c","Type":"ContainerDied","Data":"d19083fec29a51219f285f3d1e78395e26467b393ab6bf6ed80aac07d7c3e8a3"} Oct 31 01:24:31.932436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.932427 5011 scope.go:122] "RemoveContainer" containerID="91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70" Oct 31 01:24:31.945254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.945153 5011 scope.go:122] "RemoveContainer" containerID="91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70" Oct 31 01:24:31.945551 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:31.945522 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70\": container with ID starting with 91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70 not found: ID does not exist" containerID="91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70" Oct 31 01:24:31.945644 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.945566 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70"} err="failed to get container status \"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70\": rpc error: code = NotFound desc = could not find container \"91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70\": container with ID starting with 91dc1a05237984b6c66292412f95af007760fe855ce787f5f97f1c7ed8816c70 not found: ID does not exist" Oct 31 01:24:31.953110 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:31.953081 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:24:31.978101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978080 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93750db-7a67-4128-8051-6aa3aa25254c-istio-podinfo\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978101 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978102 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mllzf\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-kube-api-access-mllzf\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978112 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-credential-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978123 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-socket\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978132 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-workload-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978140 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93750db-7a67-4128-8051-6aa3aa25254c-istio-token\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978148 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93750db-7a67-4128-8051-6aa3aa25254c-istiod-ca-cert\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978156 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-data\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.978305 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.978164 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93750db-7a67-4128-8051-6aa3aa25254c-istio-envoy\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:31.985592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.985568 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 01:24:31.992714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:31.992693 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-lvxjl"] Oct 31 01:24:33.681561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:33.681526 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" path="/var/lib/kubelet/pods/b93750db-7a67-4128-8051-6aa3aa25254c/volumes" Oct 31 01:24:37.683898 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.683861 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr"] Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684275 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684288 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684305 5011 cpu_manager.go:414] "containerMap: removing container" podUID="53b938ac-d329-4abb-8c3b-5f85d67ed22c" containerName="revoke-keys" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684310 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b938ac-d329-4abb-8c3b-5f85d67ed22c" containerName="revoke-keys" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684319 5011 cpu_manager.go:414] "containerMap: removing container" podUID="aa2b21f6-1c79-4125-a034-2e86206a4fee" containerName="revoke-keys" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684325 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2b21f6-1c79-4125-a034-2e86206a4fee" containerName="revoke-keys" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684331 5011 cpu_manager.go:414] "containerMap: removing container" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684337 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684352 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" Oct 31 01:24:37.684386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684357 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" Oct 31 01:24:37.684919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684415 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74b1d56-7aca-41ab-812f-2dcc158a8134" containerName="istio-proxy" Oct 31 01:24:37.684919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684426 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93750db-7a67-4128-8051-6aa3aa25254c" containerName="istio-proxy" Oct 31 01:24:37.684919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684433 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b938ac-d329-4abb-8c3b-5f85d67ed22c" containerName="revoke-keys" Oct 31 01:24:37.684919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684440 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2b21f6-1c79-4125-a034-2e86206a4fee" containerName="revoke-keys" Oct 31 01:24:37.684919 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.684446 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="69adc0a6-30aa-4c5a-a120-656f7d569ca8" containerName="main" Oct 31 01:24:37.689046 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.689021 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.705295 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.705270 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr"] Oct 31 01:24:37.727622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.727587 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchrv\" (UniqueName: \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.727741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.727702 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.727793 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.727771 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.776639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.776614 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:37.776639 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.776630 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:24:37.777400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.777385 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:37.777558 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.777543 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:37.829230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.829194 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchrv\" (UniqueName: \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.829328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.829253 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.829328 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.829295 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.829948 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.829924 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.831827 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.831806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:37.852449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:37.852429 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchrv\" (UniqueName: \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\") pod \"maas-api-revoke-sub-f353afb8-jvtnr\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:38.003059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:38.002990 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:38.304820 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:38.304777 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr"] Oct 31 01:24:38.306321 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:24:38.306298 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa3c87a_0f3e_4670_ab59_6d2916d20622.slice/crio-3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b WatchSource:0}: Error finding container 3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b: Status 404 returned error can't find the container with id 3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b Oct 31 01:24:38.982853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:38.982766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" event={"ID":"2aa3c87a-0f3e-4670-ab59-6d2916d20622","Type":"ContainerStarted","Data":"bf98149e40f2f18f8278273201651af55c96487c0c3261674e9908780e215d6d"} Oct 31 01:24:38.982853 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:38.982819 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" event={"ID":"2aa3c87a-0f3e-4670-ab59-6d2916d20622","Type":"ContainerStarted","Data":"3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b"} Oct 31 01:24:39.023850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:39.023798 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" podStartSLOduration=2.023782386 podStartE2EDuration="2.023782386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:39.022644878 +0000 UTC m=+3947.471732511" watchObservedRunningTime="2026-10-31 01:24:39.023782386 +0000 UTC m=+3947.472870026" Oct 31 01:24:39.991562 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:39.991528 5011 generic.go:356] "Generic (PLEG): container finished" podID="2aa3c87a-0f3e-4670-ab59-6d2916d20622" containerID="bf98149e40f2f18f8278273201651af55c96487c0c3261674e9908780e215d6d" exitCode=0 Oct 31 01:24:39.991980 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:39.991611 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" event={"ID":"2aa3c87a-0f3e-4670-ab59-6d2916d20622","Type":"ContainerDied","Data":"bf98149e40f2f18f8278273201651af55c96487c0c3261674e9908780e215d6d"} Oct 31 01:24:41.135834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.135808 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:41.263478 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.263390 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\" (UniqueName: \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\") pod \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " Oct 31 01:24:41.263741 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.263721 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\") pod \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " Oct 31 01:24:41.263918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.263903 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\") pod \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\" (UID: \"2aa3c87a-0f3e-4670-ab59-6d2916d20622\") " Oct 31 01:24:41.267408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.267071 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca" pod "2aa3c87a-0f3e-4670-ab59-6d2916d20622" (UID: "2aa3c87a-0f3e-4670-ab59-6d2916d20622"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:41.272304 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.272271 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv" pod "2aa3c87a-0f3e-4670-ab59-6d2916d20622" (UID: "2aa3c87a-0f3e-4670-ab59-6d2916d20622"). InnerVolumeSpecName "kube-api-access-fchrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:41.272715 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.272682 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca" pod "2aa3c87a-0f3e-4670-ab59-6d2916d20622" (UID: "2aa3c87a-0f3e-4670-ab59-6d2916d20622"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:41.364956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.364928 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchrv\" (UniqueName: \"kubernetes.io/projected/2aa3c87a-0f3e-4670-ab59-6d2916d20622-kube-api-access-fchrv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:41.364956 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.364956 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:41.365106 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:41.364967 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa3c87a-0f3e-4670-ab59-6d2916d20622-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:42.005804 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:42.005728 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" Oct 31 01:24:42.005958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:42.005719 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr" event={"ID":"2aa3c87a-0f3e-4670-ab59-6d2916d20622","Type":"ContainerDied","Data":"3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b"} Oct 31 01:24:42.005958 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:42.005852 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c3f43b292a0bf01b359aef5661fbe92a64660fc0d56f7f7e1b020454ac3393b" Oct 31 01:24:44.226770 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.226728 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" containerID="cri-o://b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0" gracePeriod=45 Oct 31 01:24:44.238365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.238304 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" containerID="cri-o://543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438" gracePeriod=45 Oct 31 01:24:44.510110 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.510084 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 01:24:44.548256 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.548223 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 01:24:44.593779 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593741 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.593982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593805 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.593982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593835 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.593982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593862 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.593982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593916 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\" (UniqueName: \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.593982 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593948 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.593997 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594046 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594081 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\" (UniqueName: \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594110 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\" (UniqueName: \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\") pod \"447a3da0-79c8-482d-9da8-e64356f114df\" (UID: \"447a3da0-79c8-482d-9da8-e64356f114df\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594171 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594193 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.594392 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594234 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\") pod \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\" (UID: \"5a663f51-7fe6-4064-a452-4d15d4e76b3c\") " Oct 31 01:24:44.594861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594498 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.594861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594618 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.594861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594638 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.594861 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.594750 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.595630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.595337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.595630 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.595465 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.595935 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.595900 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.597004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.596976 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "kube-api-access-sbvsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:44.597159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.597136 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:44.598387 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.598347 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.598663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.598639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs" pod "5a663f51-7fe6-4064-a452-4d15d4e76b3c" (UID: "5a663f51-7fe6-4064-a452-4d15d4e76b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:44.598774 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.598740 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "kube-api-access-j7f67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:44.599462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.599430 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm" pod "447a3da0-79c8-482d-9da8-e64356f114df" (UID: "447a3da0-79c8-482d-9da8-e64356f114df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.695824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695791 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7f67\" (UniqueName: \"kubernetes.io/projected/447a3da0-79c8-482d-9da8-e64356f114df-kube-api-access-j7f67\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.695824 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695821 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695834 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695845 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvsq\" (UniqueName: \"kubernetes.io/projected/5a663f51-7fe6-4064-a452-4d15d4e76b3c-kube-api-access-sbvsq\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695855 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447a3da0-79c8-482d-9da8-e64356f114df-tls-certs\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695865 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-model-cache\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695872 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-home\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695879 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-dshm\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695886 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a663f51-7fe6-4064-a452-4d15d4e76b3c-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.696025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:44.695894 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447a3da0-79c8-482d-9da8-e64356f114df-tmp-dir\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:24:45.029611 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.029572 5011 generic.go:356] "Generic (PLEG): container finished" podID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerID="543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438" exitCode=0 Oct 31 01:24:45.029803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.029613 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" Oct 31 01:24:45.029803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.029655 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" event={"ID":"5a663f51-7fe6-4064-a452-4d15d4e76b3c","Type":"ContainerDied","Data":"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438"} Oct 31 01:24:45.029803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.029695 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f" event={"ID":"5a663f51-7fe6-4064-a452-4d15d4e76b3c","Type":"ContainerDied","Data":"b8ee921ba9ef09e9fa4a848a32b395cc756a304fcc93fe3f149acd3c5409f5d1"} Oct 31 01:24:45.029803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.029712 5011 scope.go:122] "RemoveContainer" containerID="543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438" Oct 31 01:24:45.031539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.031516 5011 generic.go:356] "Generic (PLEG): container finished" podID="447a3da0-79c8-482d-9da8-e64356f114df" containerID="b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0" exitCode=0 Oct 31 01:24:45.031674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.031573 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" event={"ID":"447a3da0-79c8-482d-9da8-e64356f114df","Type":"ContainerDied","Data":"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0"} Oct 31 01:24:45.031674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.031596 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" event={"ID":"447a3da0-79c8-482d-9da8-e64356f114df","Type":"ContainerDied","Data":"f3e9df2dffcf896bdb1c9d6addc5ce41d7894385606a888f77bfe0b85a7e12b0"} Oct 31 01:24:45.031674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.031641 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs" Oct 31 01:24:45.041259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.041122 5011 scope.go:122] "RemoveContainer" containerID="543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438" Oct 31 01:24:45.041702 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:45.041561 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438\": container with ID starting with 543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438 not found: ID does not exist" containerID="543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438" Oct 31 01:24:45.041702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.041597 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438"} err="failed to get container status \"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438\": rpc error: code = NotFound desc = could not find container \"543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438\": container with ID starting with 543c7223379f8efcedbd75b8903bc73ffec8cb7272482c01fdb2bc559b0e9438 not found: ID does not exist" Oct 31 01:24:45.041702 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.041621 5011 scope.go:122] "RemoveContainer" containerID="b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0" Oct 31 01:24:45.050606 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.050587 5011 scope.go:122] "RemoveContainer" containerID="b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0" Oct 31 01:24:45.050856 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:24:45.050837 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0\": container with ID starting with b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0 not found: ID does not exist" containerID="b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0" Oct 31 01:24:45.050923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.050862 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0"} err="failed to get container status \"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0\": rpc error: code = NotFound desc = could not find container \"b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0\": container with ID starting with b17d36da54a9cb4776cbc32ee4cf77d08e9161a1247f38b8095bd17c749701b0 not found: ID does not exist" Oct 31 01:24:45.082185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.082153 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 01:24:45.094139 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.094118 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-735123/simulator-w2-735123-kserve-67b489cd8-6577f"] Oct 31 01:24:45.120244 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.120219 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 01:24:45.126058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.126034 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-735123/premium-w2-735123-kserve-6fdd4bfc5c-rc7fs"] Oct 31 01:24:45.682458 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.682425 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447a3da0-79c8-482d-9da8-e64356f114df" path="/var/lib/kubelet/pods/447a3da0-79c8-482d-9da8-e64356f114df/volumes" Oct 31 01:24:45.682884 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:45.682869 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" path="/var/lib/kubelet/pods/5a663f51-7fe6-4064-a452-4d15d4e76b3c/volumes" Oct 31 01:24:50.785543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:50.785501 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:24:55.482640 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.482604 5011 scope.go:122] "RemoveContainer" containerID="b437bdba9939bce2618a84f7f196457b21e579949f62d36db0fee175ceac208f" Oct 31 01:24:55.492469 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.492449 5011 scope.go:122] "RemoveContainer" containerID="8469c51a6baf7c2727d15657e21a1ef3695789ca383de617f5847972c75ea5b8" Oct 31 01:24:55.502052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.502032 5011 scope.go:122] "RemoveContainer" containerID="2a2b88e877e811705bdd1c0520a1d5111dd31a9cf67f6f381b3c430f0189f3d2" Oct 31 01:24:55.510560 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.510538 5011 scope.go:122] "RemoveContainer" containerID="afda80994ad5e8595d8fdd7c4151b59f2baa1476ce26b26990d0ce19d6625a2a" Oct 31 01:24:55.518799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.518773 5011 scope.go:122] "RemoveContainer" containerID="7b47635b253b81211deb658e6cff36db7e971b3a17caf39140ed92efe1c2d150" Oct 31 01:24:55.527113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.527093 5011 scope.go:122] "RemoveContainer" containerID="47414a9fa072ac6bbf82bac61aadb931f76e2558cccf48139c52e97bcdc41999" Oct 31 01:24:55.535500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.535481 5011 scope.go:122] "RemoveContainer" containerID="7607ce3f66e43e635eab3aa635d09254bcafacfc78b882835ec2c4a653b4473d" Oct 31 01:24:55.543940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:24:55.543915 5011 scope.go:122] "RemoveContainer" containerID="872fc8d736e661a61fa7ab4e5a125c5955df5df79046c7eb7ea9eeae773a071a" Oct 31 01:25:01.204796 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:01.204758 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:25:01.946514 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:25:01.946481 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:25:31.944947 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:25:31.944913 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:25:49.031652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.031570 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t"] Oct 31 01:25:49.043457 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.043422 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj"] Oct 31 01:25:49.045609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.045579 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1210f6-62n6t"] Oct 31 01:25:49.049436 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.049405 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7ad1f2-pqwdj"] Oct 31 01:25:49.681564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.681527 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb" path="/var/lib/kubelet/pods/0d5925f8-ccaf-4dc3-b5c8-ee3426a257eb/volumes" Oct 31 01:25:49.681937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:49.681923 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a09f0b-ba6a-41b9-b668-010f56e8af44" path="/var/lib/kubelet/pods/30a09f0b-ba6a-41b9-b668-010f56e8af44/volumes" Oct 31 01:25:55.625317 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:55.625283 5011 scope.go:122] "RemoveContainer" containerID="f8c736c6a53b0977d7e3e00af2acdea602872c41f342b15ada680ce02f5c764b" Oct 31 01:25:55.634589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:55.634557 5011 scope.go:122] "RemoveContainer" containerID="8ea42091a3a33ee42eaf7aaa387ebac8cb534ba1ae9c2ca77b91fb4eebfaba19" Oct 31 01:25:55.643324 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:55.643298 5011 scope.go:122] "RemoveContainer" containerID="c6e39777c82f097c703e3a1d6adee4724101b666f0702a9a301ded6d8e946edb" Oct 31 01:25:55.651359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:25:55.651342 5011 scope.go:122] "RemoveContainer" containerID="74fedaa0034dac4f40c4266725cb6c80d63fae529f6a37f532b846cb986497b7" Oct 31 01:26:01.951627 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:26:01.951583 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:26:07.029752 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.029715 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp"] Oct 31 01:26:07.041795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.041769 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc"] Oct 31 01:26:07.043555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.043534 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed03ec3a-b68sp"] Oct 31 01:26:07.045188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.045169 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52accf06-xcltc"] Oct 31 01:26:07.681118 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.681080 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5953c647-7300-4db4-937a-11a980430014" path="/var/lib/kubelet/pods/5953c647-7300-4db4-937a-11a980430014/volumes" Oct 31 01:26:07.681509 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:07.681494 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899a2659-7305-4596-8013-882673f02051" path="/var/lib/kubelet/pods/899a2659-7305-4596-8013-882673f02051/volumes" Oct 31 01:26:16.040080 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:16.040032 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k"] Oct 31 01:26:16.042034 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:16.042007 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00903595-b5h6k"] Oct 31 01:26:17.680965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:17.680930 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4933f508-43ea-413b-99a3-53f433d53e60" path="/var/lib/kubelet/pods/4933f508-43ea-413b-99a3-53f433d53e60/volumes" Oct 31 01:26:21.664995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:21.664958 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:26:27.030003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:27.029967 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj"] Oct 31 01:26:27.041792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:27.041765 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd817dd4-fhdpj"] Oct 31 01:26:27.681225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:27.681171 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5" path="/var/lib/kubelet/pods/81a0bf4d-57c4-47b9-9bb7-5fc3eace48f5/volumes" Oct 31 01:26:31.946438 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:26:31.946400 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:26:35.802246 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802215 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p"] Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802606 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2aa3c87a-0f3e-4670-ab59-6d2916d20622" containerName="revoke-keys" Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802619 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa3c87a-0f3e-4670-ab59-6d2916d20622" containerName="revoke-keys" Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802633 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802640 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802669 5011 cpu_manager.go:414] "containerMap: removing container" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" Oct 31 01:26:35.802677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802675 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" Oct 31 01:26:35.803358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802771 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa3c87a-0f3e-4670-ab59-6d2916d20622" containerName="revoke-keys" Oct 31 01:26:35.803358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802788 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a663f51-7fe6-4064-a452-4d15d4e76b3c" containerName="main" Oct 31 01:26:35.803358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.802805 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="447a3da0-79c8-482d-9da8-e64356f114df" containerName="main" Oct 31 01:26:35.806603 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.806575 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.828437 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.828410 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p"] Oct 31 01:26:35.883419 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883388 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883427 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883456 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5k7\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-kube-api-access-kp5k7\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883592 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883527 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883660 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883795 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883766 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.883951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.883792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762c1b0e-8055-4782-b0d9-7ca81f06332b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984663 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984632 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984677 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5k7\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-kube-api-access-kp5k7\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984749 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.984816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984807 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984837 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984856 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762c1b0e-8055-4782-b0d9-7ca81f06332b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985096 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.984885 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.985200 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985389 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.985360 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.985421 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.985571 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.985645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.985627 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762c1b0e-8055-4782-b0d9-7ca81f06332b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.987647 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.987621 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:35.987890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:35.987869 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:36.008443 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.008413 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:36.008626 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.008608 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5k7\" (UniqueName: \"kubernetes.io/projected/762c1b0e-8055-4782-b0d9-7ca81f06332b-kube-api-access-kp5k7\") pod \"maas-default-gateway-openshift-default-6886b96f9-8gm8p\" (UID: \"762c1b0e-8055-4782-b0d9-7ca81f06332b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:36.125083 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.125061 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:36.447812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.447785 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p"] Oct 31 01:26:36.449498 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:26:36.449468 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762c1b0e_8055_4782_b0d9_7ca81f06332b.slice/crio-e6a2b75f0227bff35f1f564f0a23a948c14911519fa04fe57c6464411b30e4de WatchSource:0}: Error finding container e6a2b75f0227bff35f1f564f0a23a948c14911519fa04fe57c6464411b30e4de: Status 404 returned error can't find the container with id e6a2b75f0227bff35f1f564f0a23a948c14911519fa04fe57c6464411b30e4de Oct 31 01:26:36.451812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.451782 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:36.451888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.451845 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:36.451888 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.451875 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:36.749438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.749347 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" event={"ID":"762c1b0e-8055-4782-b0d9-7ca81f06332b","Type":"ContainerStarted","Data":"1d6ee052292e282526422b3430838b8f2294dd0e7459f7945990640430ccfc50"} Oct 31 01:26:36.749438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.749403 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" event={"ID":"762c1b0e-8055-4782-b0d9-7ca81f06332b","Type":"ContainerStarted","Data":"e6a2b75f0227bff35f1f564f0a23a948c14911519fa04fe57c6464411b30e4de"} Oct 31 01:26:36.796269 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:36.796197 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podStartSLOduration=1.7961782739999999 podStartE2EDuration="1.796178274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:36.794474202 +0000 UTC m=+4065.243561841" watchObservedRunningTime="2026-10-31 01:26:36.796178274 +0000 UTC m=+4065.245265916" Oct 31 01:26:37.142233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:37.134612 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:37.142233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:37.136993 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:37.142233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:37.137060 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:38.129230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:38.128588 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:38.129230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:38.128649 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:38.840231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:38.834189 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:26:39.131656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:39.131561 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:39.131928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:39.131896 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:40.127229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:40.126186 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:40.127229 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:40.126292 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:41.139231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:41.129710 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:41.139231 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:41.129790 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:42.131232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:42.128584 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:42.131232 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:42.128648 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:43.126152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:43.126106 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 01:26:43.126683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:43.126178 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 01:26:45.126579 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:45.126524 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:45.127104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:45.126652 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:45.781928 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:45.781243 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:26:46.127227 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:46.127174 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:46.127755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:46.127248 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:47.128344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:47.128302 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:47.128778 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:47.128353 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:48.169409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:48.169369 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8gm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:48.169847 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:48.169451 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" podUID="762c1b0e-8055-4782-b0d9-7ca81f06332b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:48.832726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:48.832690 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:48.833121 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:48.833095 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:48.872566 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:48.872538 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8gm8p" Oct 31 01:26:50.843302 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.843268 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr"] Oct 31 01:26:50.847027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.847006 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.869541 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.869516 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr"] Oct 31 01:26:50.943988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.943960 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944126 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944101 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944147 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252fw\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-kube-api-access-252fw\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944190 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944252 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944242 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944413 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944463 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944449 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944504 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944483 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:50.944545 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:50.944523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045068 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045105 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252fw\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-kube-api-access-252fw\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045121 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045137 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045253 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045302 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045342 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045379 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045467 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045535 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045600 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.045904 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.045730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.046090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.046073 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.047758 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.047730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.048065 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.048047 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.068698 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.068671 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.068811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.068797 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252fw\" (UniqueName: \"kubernetes.io/projected/008798e7-1bb6-4855-ba26-7f3a7dbc89ec-kube-api-access-252fw\") pod \"maas-default-gateway-openshift-default-6886b96f9-zhprr\" (UID: \"008798e7-1bb6-4855-ba26-7f3a7dbc89ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.160250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.160158 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:51.474850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.474807 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr"] Oct 31 01:26:51.474981 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:26:51.474808 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008798e7_1bb6_4855_ba26_7f3a7dbc89ec.slice/crio-d84f245e6af5c10634bbd76123727978a46ff4df3f750dc1f83394937b4bc0d9 WatchSource:0}: Error finding container d84f245e6af5c10634bbd76123727978a46ff4df3f750dc1f83394937b4bc0d9: Status 404 returned error can't find the container with id d84f245e6af5c10634bbd76123727978a46ff4df3f750dc1f83394937b4bc0d9 Oct 31 01:26:51.476976 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.476947 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:51.477044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.477012 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:51.477085 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.477042 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:26:51.850429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.850388 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" event={"ID":"008798e7-1bb6-4855-ba26-7f3a7dbc89ec","Type":"ContainerStarted","Data":"a2208ee54620e925d4308a61ee0741a9af1dc35a860c83396749debfd33242a0"} Oct 31 01:26:51.850831 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.850451 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" event={"ID":"008798e7-1bb6-4855-ba26-7f3a7dbc89ec","Type":"ContainerStarted","Data":"d84f245e6af5c10634bbd76123727978a46ff4df3f750dc1f83394937b4bc0d9"} Oct 31 01:26:51.909638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:51.909582 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podStartSLOduration=1.9095696850000001 podStartE2EDuration="1.909569685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:51.907921087 +0000 UTC m=+4080.357008725" watchObservedRunningTime="2026-10-31 01:26:51.909569685 +0000 UTC m=+4080.358657323" Oct 31 01:26:52.160766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:52.160672 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:52.162393 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:52.162364 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:52.162520 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:52.162430 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:53.161482 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:53.161441 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:53.161883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:53.161501 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:54.160879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:54.160837 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:54.161088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:54.160912 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:55.160737 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.160705 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:55.161131 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.160755 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:55.690104 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.690054 5011 scope.go:122] "RemoveContainer" containerID="dad455066f7d02b7f96fafc2f6f21b8eeb228e03d66f961127a0e8cc946fd442" Oct 31 01:26:55.699704 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.699681 5011 scope.go:122] "RemoveContainer" containerID="40826f583c0799ddfb59eae5292a29cf770441ecc08b20c4848f38a2795c9c67" Oct 31 01:26:55.708087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.708068 5011 scope.go:122] "RemoveContainer" containerID="c9cd41ded8f9db3d0d2a33a33d2ad803f750626dae6e8b350c2026d1b9676bfb" Oct 31 01:26:55.716343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.716324 5011 scope.go:122] "RemoveContainer" containerID="0d8c5314ba35e3e08e09ed5b0f02334a6b74d7adc50d82a5b3533fa75da2c77a" Oct 31 01:26:55.748468 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.748449 5011 scope.go:122] "RemoveContainer" containerID="1e45626161a57ad72af3f09dc74b68afc17b5c9790a4bcec075f8f91b320657d" Oct 31 01:26:55.757071 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:55.757049 5011 scope.go:122] "RemoveContainer" containerID="1a4abfb0f734f4378551bbea5f90bca66f00f5fc520620a97625d8b946e03fd3" Oct 31 01:26:56.160677 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:56.160642 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:56.160863 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:56.160707 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:57.161358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:57.161317 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:57.161755 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:57.161373 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:58.160675 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:58.160642 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zhprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 31 01:26:58.160859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:58.160696 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" podUID="008798e7-1bb6-4855-ba26-7f3a7dbc89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 31 01:26:59.589913 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:59.589863 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:59.590506 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:59.590479 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:26:59.591321 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:26:59.591301 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zhprr" Oct 31 01:27:01.947003 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:27:01.946959 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:27:01.995772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:01.995733 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:27:17.770174 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:17.770134 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:27:31.945737 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:27:31.945701 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:27:41.030467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.030430 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr"] Oct 31 01:27:41.042960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.042919 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns"] Oct 31 01:27:41.045237 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.045195 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448d4daa-hsfcr"] Oct 31 01:27:41.047037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.047008 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523fc951-v4nns"] Oct 31 01:27:41.278816 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.278780 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:27:41.684100 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.684060 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e45ad7-1498-4d79-96ab-139941c2dfa9" path="/var/lib/kubelet/pods/04e45ad7-1498-4d79-96ab-139941c2dfa9/volumes" Oct 31 01:27:41.684699 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:41.684680 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a" path="/var/lib/kubelet/pods/ee054917-39b6-4e9f-bbce-f3ef0f2a5f0a/volumes" Oct 31 01:27:51.710059 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:51.710026 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:27:55.800693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.800660 5011 scope.go:122] "RemoveContainer" containerID="290e9a7faf785b9b9519319027587b0a689771fcfdf50b563b2c2858f62da00d" Oct 31 01:27:55.809947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.809923 5011 scope.go:122] "RemoveContainer" containerID="cba557e711264cbd549ebdafe9190bc6b71c0e350106b2a2037c9a2518ce19b3" Oct 31 01:27:55.818160 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.818139 5011 scope.go:122] "RemoveContainer" containerID="e0e0eddf692f9e0f714590ee305baddeabc5aff83d84fa9617b3491c293d5a19" Oct 31 01:27:55.826264 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.826241 5011 scope.go:122] "RemoveContainer" containerID="9cd005b30ea918464392e81e8184c0666c8bf65c638989698fef588bc5774f5c" Oct 31 01:27:55.837678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.837648 5011 scope.go:122] "RemoveContainer" containerID="9f3a73cc501880e93e7186ed7bbdc76ba8ef822665935da050ceb5ff24bb40d5" Oct 31 01:27:55.846535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.846513 5011 scope.go:122] "RemoveContainer" containerID="b40365de0f93e74b63da8ccb8542d178c73e4658a56deb0f9755f825e9c11270" Oct 31 01:27:55.856414 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:55.856396 5011 scope.go:122] "RemoveContainer" containerID="bf6c76c236162a617d4cab5692ea289c019143fd6ffdbac76ea1a0021acb91b1" Oct 31 01:27:56.414626 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:27:56.414589 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:28:01.945838 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:28:01.945794 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:28:13.711975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:13.711939 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:28:19.443781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:19.443742 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:28:31.029438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:31.029398 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd"] Oct 31 01:28:31.041689 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:31.041660 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c5f78e-97ktd"] Oct 31 01:28:31.681024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:31.680992 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42eee6bc-a7a4-412d-9974-7e13fbbe49b5" path="/var/lib/kubelet/pods/42eee6bc-a7a4-412d-9974-7e13fbbe49b5/volumes" Oct 31 01:28:31.945505 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:28:31.945421 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:28:35.412076 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:35.412003 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:28:41.488766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:41.488726 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:28:52.775525 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:52.775498 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:28:52.797959 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:52.797937 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:28:55.908638 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.908604 5011 scope.go:122] "RemoveContainer" containerID="03bed4c097c154434620f1e53bdb22680b3c0a0d3150060fbb91abe5c37d2226" Oct 31 01:28:55.918873 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.918847 5011 scope.go:122] "RemoveContainer" containerID="c5f72c73a79e9a0e154e6e86bfe75d5e8304771de399ad8de5bd1c0214aedf2b" Oct 31 01:28:55.927193 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.927174 5011 scope.go:122] "RemoveContainer" containerID="af279d498c54d5a51853d582532400b2cc69a20973406b43d207497b81d73529" Oct 31 01:28:55.935869 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.935843 5011 scope.go:122] "RemoveContainer" containerID="a88f2adc45e45e31bb1271715e318f82ed7cc506a03b8588d498885531dc415a" Oct 31 01:28:55.944222 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.944176 5011 scope.go:122] "RemoveContainer" containerID="fd931abb9120fa1600b72c5b765f983f085f8a82a9afe306556566f53e595152" Oct 31 01:28:55.955682 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.955661 5011 scope.go:122] "RemoveContainer" containerID="dea4157af9b7042bb9ef0aa9e144ba05a6fad53f3c1d87bf31de6cb87b6a8758" Oct 31 01:28:55.965923 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:55.965897 5011 scope.go:122] "RemoveContainer" containerID="195510f388609b9d63f1a53f439305e2a5c15ee675349e8eef69ad7a0697a75f" Oct 31 01:28:56.676292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:28:56.676257 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:01.997300 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:29:01.997255 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:29:01.999810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:01.999782 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:09.038643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:09.038610 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf"] Oct 31 01:29:09.050599 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:09.050552 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca66de-vr8qf"] Oct 31 01:29:09.681329 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:09.681297 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2b21f6-1c79-4125-a034-2e86206a4fee" path="/var/lib/kubelet/pods/aa2b21f6-1c79-4125-a034-2e86206a4fee/volumes" Oct 31 01:29:16.784048 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:16.784007 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:27.474757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:27.474720 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:32.008022 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:29:32.007987 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:29:33.393850 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:33.393817 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:42.039334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:42.039294 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr"] Oct 31 01:29:42.050718 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:42.050686 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353afb8-jvtnr"] Oct 31 01:29:43.681832 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:43.681796 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa3c87a-0f3e-4670-ab59-6d2916d20622" path="/var/lib/kubelet/pods/2aa3c87a-0f3e-4670-ab59-6d2916d20622/volumes" Oct 31 01:29:49.810666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:49.810628 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:55.286787 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:55.286748 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:29:56.012099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:56.012062 5011 scope.go:122] "RemoveContainer" containerID="c2992f62190f688f43830cd6c1c21d6d8fcab507945e6897bb2c1b1fad02d695" Oct 31 01:29:56.021276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:56.021256 5011 scope.go:122] "RemoveContainer" containerID="1669bc798e3982dd4f8cd4bd42bd870c1ef730e9c2a5a07356cf937219c4489f" Oct 31 01:29:56.030274 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:56.030255 5011 scope.go:122] "RemoveContainer" containerID="4fff0c033ede42c04d604376465050bd574571ed7ede27d8d3a22fd978c91b04" Oct 31 01:29:56.039750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:56.039731 5011 scope.go:122] "RemoveContainer" containerID="265385f8010b84a5e63c0163f2ab8cac3f7fca9f9bedfe6295021c1f29b4cdaf" Oct 31 01:29:56.047823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:29:56.047800 5011 scope.go:122] "RemoveContainer" containerID="d0b6cc9cc8a56b33b079ed443f72cc22fb241236facdbf775d22d4d1c5fc8d2d" Oct 31 01:30:01.948763 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:30:01.948711 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:30:04.994494 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:04.994414 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:30:11.632645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.632608 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6"] Oct 31 01:30:11.636159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.636139 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.653385 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.653358 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6"] Oct 31 01:30:11.662753 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.662716 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:11.662871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.662779 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:30:11.662871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.662793 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:11.662871 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.662788 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:11.705142 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.705114 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdcd\" (UniqueName: \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.705294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.705256 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.705294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.705284 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.806648 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.806613 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdcd\" (UniqueName: \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.806782 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.806706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.806782 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.806736 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.807424 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.807399 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.808977 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.808960 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.829727 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.829700 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdcd\" (UniqueName: \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\") pod \"maas-api-revoke-sub-e2fc9c5b-6wcv6\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:11.948598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:11.948525 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:12.245167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:12.245133 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6"] Oct 31 01:30:12.246090 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:30:12.246062 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335df35d_d221_4d14_a85d_b173097d2c72.slice/crio-cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c WatchSource:0}: Error finding container cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c: Status 404 returned error can't find the container with id cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c Oct 31 01:30:13.172757 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:13.172715 5011 generic.go:356] "Generic (PLEG): container finished" podID="335df35d-d221-4d14-a85d-b173097d2c72" containerID="c44016c58128e85936b010fec6cc9d684b5646a8e49bc5767e4ef6c29e39fca8" exitCode=0 Oct 31 01:30:13.173154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:13.172753 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" event={"ID":"335df35d-d221-4d14-a85d-b173097d2c72","Type":"ContainerDied","Data":"c44016c58128e85936b010fec6cc9d684b5646a8e49bc5767e4ef6c29e39fca8"} Oct 31 01:30:13.173154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:13.172853 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" event={"ID":"335df35d-d221-4d14-a85d-b173097d2c72","Type":"ContainerStarted","Data":"cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c"} Oct 31 01:30:14.321567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.321540 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:14.428916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.428842 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\" (UniqueName: \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\") pod \"335df35d-d221-4d14-a85d-b173097d2c72\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " Oct 31 01:30:14.428916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.428888 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\") pod \"335df35d-d221-4d14-a85d-b173097d2c72\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " Oct 31 01:30:14.429125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.428924 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\") pod \"335df35d-d221-4d14-a85d-b173097d2c72\" (UID: \"335df35d-d221-4d14-a85d-b173097d2c72\") " Oct 31 01:30:14.429399 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.429372 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca" pod "335df35d-d221-4d14-a85d-b173097d2c72" (UID: "335df35d-d221-4d14-a85d-b173097d2c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:14.430995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.430966 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca" pod "335df35d-d221-4d14-a85d-b173097d2c72" (UID: "335df35d-d221-4d14-a85d-b173097d2c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:14.430995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.430980 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd" pod "335df35d-d221-4d14-a85d-b173097d2c72" (UID: "335df35d-d221-4d14-a85d-b173097d2c72"). InnerVolumeSpecName "kube-api-access-cqdcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:14.530261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.530235 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdcd\" (UniqueName: \"kubernetes.io/projected/335df35d-d221-4d14-a85d-b173097d2c72-kube-api-access-cqdcd\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:30:14.530261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.530258 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:30:14.530412 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:14.530268 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335df35d-d221-4d14-a85d-b173097d2c72-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:30:15.184215 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:15.184184 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" Oct 31 01:30:15.184444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:15.184230 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6" event={"ID":"335df35d-d221-4d14-a85d-b173097d2c72","Type":"ContainerDied","Data":"cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c"} Oct 31 01:30:15.184444 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:15.184274 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae715954f43fac5a7042e6e7d028f8f250bf2b28e1e9fde68d6e8b79d7b8d1c" Oct 31 01:30:23.003340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:23.003301 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:30:31.945462 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:30:31.945424 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:30:33.309990 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:33.309957 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:30:53.989723 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:53.989680 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:30:56.092025 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:56.091991 5011 scope.go:122] "RemoveContainer" containerID="0413fe374ff88169023040b8b5ba28c9966eb4afdc198094da2c32af2b7be1f4" Oct 31 01:30:56.106495 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:56.106474 5011 scope.go:122] "RemoveContainer" containerID="bf98149e40f2f18f8278273201651af55c96487c0c3261674e9908780e215d6d" Oct 31 01:30:56.117907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:56.117889 5011 scope.go:122] "RemoveContainer" containerID="c30950f24af87f1407ff0fb9a0be2230a3a0cc997e6056cbc1100ba46b6da1dc" Oct 31 01:30:59.918240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:30:59.918189 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:31:01.949082 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:31:01.949031 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:31:13.314359 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:13.314318 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:31:27.580632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:27.580592 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:31:31.949177 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:31:31.949139 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:31:43.774184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:43.774102 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:31:57.706698 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.706661 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8"] Oct 31 01:31:57.707116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.707047 5011 cpu_manager.go:414] "containerMap: removing container" podUID="335df35d-d221-4d14-a85d-b173097d2c72" containerName="revoke-keys" Oct 31 01:31:57.707116 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.707059 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335df35d-d221-4d14-a85d-b173097d2c72" containerName="revoke-keys" Oct 31 01:31:57.707197 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.707140 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="335df35d-d221-4d14-a85d-b173097d2c72" containerName="revoke-keys" Oct 31 01:31:57.709749 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.709732 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.738513 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.738488 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:31:57.738688 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.738500 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:31:57.738688 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.738518 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:31:57.738900 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.738603 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:31:57.740798 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.740777 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8"] Oct 31 01:31:57.753150 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.753124 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksxv\" (UniqueName: \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.753299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.753184 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.753299 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.753245 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.854163 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.854136 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.854346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.854180 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.854346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.854327 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksxv\" (UniqueName: \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.854782 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.854756 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.856653 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.856630 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:57.877705 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:57.877681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksxv\" (UniqueName: \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\") pod \"maas-api-revoke-sub-1a92947e-75vb8\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:58.022474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:58.022389 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:31:58.321948 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:31:58.321917 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8480424_a1b8_47cc_83b8_bd4bd55fce0d.slice/crio-977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8 WatchSource:0}: Error finding container 977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8: Status 404 returned error can't find the container with id 977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8 Oct 31 01:31:58.323075 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:58.323045 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8"] Oct 31 01:31:58.752062 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:58.752024 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" event={"ID":"a8480424-a1b8-47cc-83b8-bd4bd55fce0d","Type":"ContainerStarted","Data":"a6b13892857df859f894bb885383094996bf48080215e3aade3191aa1d26e400"} Oct 31 01:31:58.752454 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:58.752081 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" event={"ID":"a8480424-a1b8-47cc-83b8-bd4bd55fce0d","Type":"ContainerStarted","Data":"977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8"} Oct 31 01:31:58.793726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:58.793684 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" podStartSLOduration=1.7936700669999999 podStartE2EDuration="1.793670067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:58.792587432 +0000 UTC m=+4387.241675070" watchObservedRunningTime="2026-10-31 01:31:58.793670067 +0000 UTC m=+4387.242757706" Oct 31 01:31:59.758409 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:59.758369 5011 generic.go:356] "Generic (PLEG): container finished" podID="a8480424-a1b8-47cc-83b8-bd4bd55fce0d" containerID="a6b13892857df859f894bb885383094996bf48080215e3aade3191aa1d26e400" exitCode=0 Oct 31 01:31:59.758810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:31:59.758450 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" event={"ID":"a8480424-a1b8-47cc-83b8-bd4bd55fce0d","Type":"ContainerDied","Data":"a6b13892857df859f894bb885383094996bf48080215e3aade3191aa1d26e400"} Oct 31 01:32:00.906089 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.906063 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:32:00.982182 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.982156 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\" (UniqueName: \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\") pod \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " Oct 31 01:32:00.982338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.982193 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\") pod \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " Oct 31 01:32:00.982338 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.982257 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\") pod \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\" (UID: \"a8480424-a1b8-47cc-83b8-bd4bd55fce0d\") " Oct 31 01:32:00.982628 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.982597 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca" pod "a8480424-a1b8-47cc-83b8-bd4bd55fce0d" (UID: "a8480424-a1b8-47cc-83b8-bd4bd55fce0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:00.984297 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.984275 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv" pod "a8480424-a1b8-47cc-83b8-bd4bd55fce0d" (UID: "a8480424-a1b8-47cc-83b8-bd4bd55fce0d"). InnerVolumeSpecName "kube-api-access-wksxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:00.984531 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:00.984501 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca" pod "a8480424-a1b8-47cc-83b8-bd4bd55fce0d" (UID: "a8480424-a1b8-47cc-83b8-bd4bd55fce0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:01.082961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.082938 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wksxv\" (UniqueName: \"kubernetes.io/projected/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-kube-api-access-wksxv\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:01.082961 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.082961 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:01.083115 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.082971 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8480424-a1b8-47cc-83b8-bd4bd55fce0d-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:01.769138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.769107 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" Oct 31 01:32:01.769309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.769106 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8" event={"ID":"a8480424-a1b8-47cc-83b8-bd4bd55fce0d","Type":"ContainerDied","Data":"977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8"} Oct 31 01:32:01.769309 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:01.769243 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="977c0892d3f0054b3cf3c1795260f836293389818e25ed4e6517cfb08f6edff8" Oct 31 01:32:01.946069 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:32:01.945988 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:32:09.088547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.088510 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4"] Oct 31 01:32:09.088922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.088903 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a8480424-a1b8-47cc-83b8-bd4bd55fce0d" containerName="revoke-keys" Oct 31 01:32:09.088962 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.088928 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8480424-a1b8-47cc-83b8-bd4bd55fce0d" containerName="revoke-keys" Oct 31 01:32:09.089067 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.089053 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8480424-a1b8-47cc-83b8-bd4bd55fce0d" containerName="revoke-keys" Oct 31 01:32:09.092355 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.092336 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.111012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.110987 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4"] Oct 31 01:32:09.120523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.120497 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:32:09.120523 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.120507 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:32:09.152907 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.152884 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.153037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.152917 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.153037 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.152947 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzrr\" (UniqueName: \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.188922 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.188896 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:32:09.189027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.188998 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:32:09.253612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.253586 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.253740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.253621 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.253740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.253662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzrr\" (UniqueName: \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.254314 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.254293 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.256072 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.256051 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.278950 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.278925 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzrr\" (UniqueName: \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.405240 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.405157 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:09.705346 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.705314 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4"] Oct 31 01:32:09.705579 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:32:09.705550 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24987681_9db6_4dab_9a78_3c66dad9deca.slice/crio-9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f WatchSource:0}: Error finding container 9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f: Status 404 returned error can't find the container with id 9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f Oct 31 01:32:09.810860 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:09.810820 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" event={"ID":"24987681-9db6-4dab-9a78-3c66dad9deca","Type":"ContainerStarted","Data":"9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f"} Oct 31 01:32:10.200460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:10.200414 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:32:10.818799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:10.818755 5011 generic.go:356] "Generic (PLEG): container finished" podID="24987681-9db6-4dab-9a78-3c66dad9deca" containerID="e7a5d5915badf3cefce1439d385a9d5bc8e65ddc7032c9de3bfe58a5192802e9" exitCode=0 Oct 31 01:32:10.818799 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:10.818799 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" event={"ID":"24987681-9db6-4dab-9a78-3c66dad9deca","Type":"ContainerDied","Data":"e7a5d5915badf3cefce1439d385a9d5bc8e65ddc7032c9de3bfe58a5192802e9"} Oct 31 01:32:11.961806 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:11.961775 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:12.078882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.078811 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\") pod \"24987681-9db6-4dab-9a78-3c66dad9deca\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " Oct 31 01:32:12.078882 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.078880 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\") pod \"24987681-9db6-4dab-9a78-3c66dad9deca\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " Oct 31 01:32:12.079093 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.078919 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\" (UniqueName: \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\") pod \"24987681-9db6-4dab-9a78-3c66dad9deca\" (UID: \"24987681-9db6-4dab-9a78-3c66dad9deca\") " Oct 31 01:32:12.079288 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.079261 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca" pod "24987681-9db6-4dab-9a78-3c66dad9deca" (UID: "24987681-9db6-4dab-9a78-3c66dad9deca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:12.081016 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.080988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca" pod "24987681-9db6-4dab-9a78-3c66dad9deca" (UID: "24987681-9db6-4dab-9a78-3c66dad9deca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:12.081167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.081142 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr" pod "24987681-9db6-4dab-9a78-3c66dad9deca" (UID: "24987681-9db6-4dab-9a78-3c66dad9deca"). InnerVolumeSpecName "kube-api-access-xpzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:12.179834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.179805 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:12.179834 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.179833 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24987681-9db6-4dab-9a78-3c66dad9deca-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:12.179967 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.179842 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzrr\" (UniqueName: \"kubernetes.io/projected/24987681-9db6-4dab-9a78-3c66dad9deca-kube-api-access-xpzrr\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:12.831810 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.831770 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" event={"ID":"24987681-9db6-4dab-9a78-3c66dad9deca","Type":"ContainerDied","Data":"9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f"} Oct 31 01:32:12.832028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.831833 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c08fb2326154c64b62e3ce24cb4e733f1d0890f6f4c87c512e649dc918bef3f" Oct 31 01:32:12.832028 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:12.831784 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4" Oct 31 01:32:22.957783 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:22.957747 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4"] Oct 31 01:32:22.970157 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:22.970122 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e8e612-5zsz4"] Oct 31 01:32:23.682041 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:23.682005 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24987681-9db6-4dab-9a78-3c66dad9deca" path="/var/lib/kubelet/pods/24987681-9db6-4dab-9a78-3c66dad9deca/volumes" Oct 31 01:32:25.016908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:25.016872 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:32:31.946102 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:32:31.946061 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:32:43.295273 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:43.295229 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:32:49.391949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:49.391909 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:32:56.437515 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.437477 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 01:32:56.437936 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.437691 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" podUID="552616f2-aac8-4b53-9eee-05986d8b9db7" containerName="manager" containerID="cri-o://27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96" gracePeriod=10 Oct 31 01:32:56.703660 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.703601 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 01:32:56.812500 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.812464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\") pod \"552616f2-aac8-4b53-9eee-05986d8b9db7\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " Oct 31 01:32:56.812664 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.812506 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\" (UniqueName: \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\") pod \"552616f2-aac8-4b53-9eee-05986d8b9db7\" (UID: \"552616f2-aac8-4b53-9eee-05986d8b9db7\") " Oct 31 01:32:56.813027 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.812992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume" pod "552616f2-aac8-4b53-9eee-05986d8b9db7" (UID: "552616f2-aac8-4b53-9eee-05986d8b9db7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:56.814905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.814872 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g" pod "552616f2-aac8-4b53-9eee-05986d8b9db7" (UID: "552616f2-aac8-4b53-9eee-05986d8b9db7"). InnerVolumeSpecName "kube-api-access-jr24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:56.913742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.913706 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr24g\" (UniqueName: \"kubernetes.io/projected/552616f2-aac8-4b53-9eee-05986d8b9db7-kube-api-access-jr24g\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:56.913742 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:56.913739 5011 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/552616f2-aac8-4b53-9eee-05986d8b9db7-extensions-socket-volume\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:32:57.078846 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.078811 5011 generic.go:356] "Generic (PLEG): container finished" podID="552616f2-aac8-4b53-9eee-05986d8b9db7" containerID="27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96" exitCode=0 Oct 31 01:32:57.079004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.078859 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" Oct 31 01:32:57.079004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.078900 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" event={"ID":"552616f2-aac8-4b53-9eee-05986d8b9db7","Type":"ContainerDied","Data":"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96"} Oct 31 01:32:57.079004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.078955 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp" event={"ID":"552616f2-aac8-4b53-9eee-05986d8b9db7","Type":"ContainerDied","Data":"fa6af5d8e3409c09dfb2bb293489d995d62778a07ce36c2998c967be7bc993b7"} Oct 31 01:32:57.079004 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.078978 5011 scope.go:122] "RemoveContainer" containerID="27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96" Oct 31 01:32:57.092136 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.092121 5011 scope.go:122] "RemoveContainer" containerID="27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96" Oct 31 01:32:57.092422 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:32:57.092403 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96\": container with ID starting with 27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96 not found: ID does not exist" containerID="27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96" Oct 31 01:32:57.092487 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.092432 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96"} err="failed to get container status \"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96\": rpc error: code = NotFound desc = could not find container \"27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96\": container with ID starting with 27b05685b943b5fe93cb5e2dd81501aa3af799f4da1bd30edc02aabb92747c96 not found: ID does not exist" Oct 31 01:32:57.131184 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.131159 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 01:32:57.143347 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.143323 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-55xxp"] Oct 31 01:32:57.681090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:32:57.681058 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552616f2-aac8-4b53-9eee-05986d8b9db7" path="/var/lib/kubelet/pods/552616f2-aac8-4b53-9eee-05986d8b9db7/volumes" Oct 31 01:33:01.946638 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:33:01.946606 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:33:31.946019 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:33:31.945980 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:33:52.809188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:52.809162 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:33:52.832631 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:52.832606 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:33:57.521696 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.521661 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr"] Oct 31 01:33:57.522123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522108 5011 cpu_manager.go:414] "containerMap: removing container" podUID="552616f2-aac8-4b53-9eee-05986d8b9db7" containerName="manager" Oct 31 01:33:57.522169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522124 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552616f2-aac8-4b53-9eee-05986d8b9db7" containerName="manager" Oct 31 01:33:57.522169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522152 5011 cpu_manager.go:414] "containerMap: removing container" podUID="24987681-9db6-4dab-9a78-3c66dad9deca" containerName="revoke-keys" Oct 31 01:33:57.522169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522158 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24987681-9db6-4dab-9a78-3c66dad9deca" containerName="revoke-keys" Oct 31 01:33:57.522294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522237 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="24987681-9db6-4dab-9a78-3c66dad9deca" containerName="revoke-keys" Oct 31 01:33:57.522294 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.522250 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="552616f2-aac8-4b53-9eee-05986d8b9db7" containerName="manager" Oct 31 01:33:57.526548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.526528 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.537552 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.537528 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr"] Oct 31 01:33:57.547622 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.547600 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hbhng\"" Oct 31 01:33:57.589156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.589130 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjh56\" (UniqueName: \"kubernetes.io/projected/590fb320-40cf-438a-a1d3-7d718bb35d18-kube-api-access-zjh56\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.589275 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.589188 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590fb320-40cf-438a-a1d3-7d718bb35d18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.690621 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.690592 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjh56\" (UniqueName: \"kubernetes.io/projected/590fb320-40cf-438a-a1d3-7d718bb35d18-kube-api-access-zjh56\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.690740 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.690646 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590fb320-40cf-438a-a1d3-7d718bb35d18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.690953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.690938 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590fb320-40cf-438a-a1d3-7d718bb35d18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.714343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.714321 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjh56\" (UniqueName: \"kubernetes.io/projected/590fb320-40cf-438a-a1d3-7d718bb35d18-kube-api-access-zjh56\") pod \"kuadrant-operator-controller-manager-678d87766f-nnkvr\" (UID: \"590fb320-40cf-438a-a1d3-7d718bb35d18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:57.840310 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:57.840282 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:33:58.179598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:58.179565 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr"] Oct 31 01:33:58.180084 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:33:58.180052 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590fb320_40cf_438a_a1d3_7d718bb35d18.slice/crio-ee13b0afedb16715a33b2dbc2f76875c4f124f44a08b8152c1fb6b56b15a7df4 WatchSource:0}: Error finding container ee13b0afedb16715a33b2dbc2f76875c4f124f44a08b8152c1fb6b56b15a7df4: Status 404 returned error can't find the container with id ee13b0afedb16715a33b2dbc2f76875c4f124f44a08b8152c1fb6b56b15a7df4 Oct 31 01:33:58.398408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:58.398370 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" event={"ID":"590fb320-40cf-438a-a1d3-7d718bb35d18","Type":"ContainerStarted","Data":"b618f6827bace702b45f7ff76988f677a72ebee2ecab15da4189ad38d821edfb"} Oct 31 01:33:58.398589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:58.398424 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" event={"ID":"590fb320-40cf-438a-a1d3-7d718bb35d18","Type":"ContainerStarted","Data":"ee13b0afedb16715a33b2dbc2f76875c4f124f44a08b8152c1fb6b56b15a7df4"} Oct 31 01:33:58.398589 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:33:58.398451 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:34:01.945984 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:34:01.945949 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:34:09.407260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:09.407203 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" Oct 31 01:34:09.475747 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:09.475695 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nnkvr" podStartSLOduration=12.475681217 podStartE2EDuration="12.475681217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:58.454402305 +0000 UTC m=+4506.903489943" watchObservedRunningTime="2026-10-31 01:34:09.475681217 +0000 UTC m=+4517.924768855" Oct 31 01:34:16.319470 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:16.319435 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:34:31.947379 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:34:31.947336 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:34:32.019643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:32.019611 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:34:37.278286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:37.277752 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:34:53.475916 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:34:53.475880 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:35:01.946771 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:35:01.946727 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:35:03.343113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:03.343073 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:35:15.045181 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:15.045141 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6"] Oct 31 01:35:15.056854 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:15.056818 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6wcv6"] Oct 31 01:35:15.682165 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:15.682121 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335df35d-d221-4d14-a85d-b173097d2c72" path="/var/lib/kubelet/pods/335df35d-d221-4d14-a85d-b173097d2c72/volumes" Oct 31 01:35:19.732334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:19.732295 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:35:31.947123 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:35:31.947082 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:35:33.700564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:33.700529 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:35:45.860564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.860532 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn"] Oct 31 01:35:45.864123 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.864101 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:45.884598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.884572 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn"] Oct 31 01:35:45.893369 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.893343 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:35:45.893467 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.893442 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:35:45.896156 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.896136 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:35:45.896259 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.896155 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:35:45.900503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.900484 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jr9b\" (UniqueName: \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:45.900569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.900531 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:45.900569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:45.900559 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.001561 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.001532 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jr9b\" (UniqueName: \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.001714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.001580 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.001714 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.001695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.002132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.002112 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.004051 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.004030 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.026441 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.026411 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jr9b\" (UniqueName: \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\") pod \"maas-api-revoke-sub-c8138f35-xqlhn\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.176738 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.176664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:46.478813 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.478782 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn"] Oct 31 01:35:46.479964 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:35:46.479935 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395fd545_c908_4206_a7cd_540ca932a30c.slice/crio-3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4 WatchSource:0}: Error finding container 3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4: Status 404 returned error can't find the container with id 3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4 Oct 31 01:35:46.980446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.980349 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" event={"ID":"395fd545-c908-4206-a7cd-540ca932a30c","Type":"ContainerStarted","Data":"ec0d25e9cfe6c177bc631fc50ed11566b01500873e97e8909327251d5e013583"} Oct 31 01:35:46.980446 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:46.980407 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" event={"ID":"395fd545-c908-4206-a7cd-540ca932a30c","Type":"ContainerStarted","Data":"3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4"} Oct 31 01:35:47.022125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:47.022071 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" podStartSLOduration=2.022057396 podStartE2EDuration="2.022057396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:47.020906554 +0000 UTC m=+4615.469994193" watchObservedRunningTime="2026-10-31 01:35:47.022057396 +0000 UTC m=+4615.471145035" Oct 31 01:35:47.996649 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:47.996607 5011 generic.go:356] "Generic (PLEG): container finished" podID="395fd545-c908-4206-a7cd-540ca932a30c" containerID="ec0d25e9cfe6c177bc631fc50ed11566b01500873e97e8909327251d5e013583" exitCode=0 Oct 31 01:35:47.997087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:47.996688 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" event={"ID":"395fd545-c908-4206-a7cd-540ca932a30c","Type":"ContainerDied","Data":"ec0d25e9cfe6c177bc631fc50ed11566b01500873e97e8909327251d5e013583"} Oct 31 01:35:49.138091 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.138065 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:49.233047 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.233017 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\" (UniqueName: \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\") pod \"395fd545-c908-4206-a7cd-540ca932a30c\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " Oct 31 01:35:49.233260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.233101 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\") pod \"395fd545-c908-4206-a7cd-540ca932a30c\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " Oct 31 01:35:49.233260 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.233130 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\") pod \"395fd545-c908-4206-a7cd-540ca932a30c\" (UID: \"395fd545-c908-4206-a7cd-540ca932a30c\") " Oct 31 01:35:49.233521 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.233496 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca" pod "395fd545-c908-4206-a7cd-540ca932a30c" (UID: "395fd545-c908-4206-a7cd-540ca932a30c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:35:49.235216 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.235182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca" pod "395fd545-c908-4206-a7cd-540ca932a30c" (UID: "395fd545-c908-4206-a7cd-540ca932a30c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:35:49.235292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.235239 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b" pod "395fd545-c908-4206-a7cd-540ca932a30c" (UID: "395fd545-c908-4206-a7cd-540ca932a30c"). InnerVolumeSpecName "kube-api-access-7jr9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:35:49.334337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.334306 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:35:49.334337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.334332 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395fd545-c908-4206-a7cd-540ca932a30c-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:35:49.334337 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:49.334341 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jr9b\" (UniqueName: \"kubernetes.io/projected/395fd545-c908-4206-a7cd-540ca932a30c-kube-api-access-7jr9b\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:35:50.008683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:50.008594 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" Oct 31 01:35:50.008683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:50.008626 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn" event={"ID":"395fd545-c908-4206-a7cd-540ca932a30c","Type":"ContainerDied","Data":"3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4"} Oct 31 01:35:50.008683 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:50.008676 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c830d60754096d4554157bd5fc2064053b09511fd908fe75a13349fec3f9cc4" Oct 31 01:35:57.013102 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.013065 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7"] Oct 31 01:35:57.013537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.013487 5011 cpu_manager.go:414] "containerMap: removing container" podUID="395fd545-c908-4206-a7cd-540ca932a30c" containerName="revoke-keys" Oct 31 01:35:57.013537 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.013502 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395fd545-c908-4206-a7cd-540ca932a30c" containerName="revoke-keys" Oct 31 01:35:57.013619 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.013594 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="395fd545-c908-4206-a7cd-540ca932a30c" containerName="revoke-keys" Oct 31 01:35:57.016905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.016884 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.035137 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.035112 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7"] Oct 31 01:35:57.046943 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.046916 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:35:57.047111 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.047092 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:35:57.047186 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.047118 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:35:57.047186 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.047142 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:35:57.104190 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.104163 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.104349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.104195 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.104349 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.104328 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hs7\" (UniqueName: \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.204899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.204866 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.204899 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.204901 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.205082 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.204934 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hs7\" (UniqueName: \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.205554 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.205522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.207236 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.207196 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.228896 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.228867 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hs7\" (UniqueName: \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.330391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.330366 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:35:57.416336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.416300 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:35:57.633194 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:35:57.633156 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f8616e_d414_4f94_ace1_08db66347203.slice/crio-a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b WatchSource:0}: Error finding container a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b: Status 404 returned error can't find the container with id a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b Oct 31 01:35:57.633643 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:57.633613 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7"] Oct 31 01:35:58.057803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:58.057767 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" event={"ID":"53f8616e-d414-4f94-ace1-08db66347203","Type":"ContainerStarted","Data":"1babd637282a001e02b7d3601b9d7fdd6a4a01b7c8781a33290c05921f2eb0d0"} Oct 31 01:35:58.058168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:58.057831 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" event={"ID":"53f8616e-d414-4f94-ace1-08db66347203","Type":"ContainerStarted","Data":"a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b"} Oct 31 01:35:58.106667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:58.106617 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" podStartSLOduration=2.106602926 podStartE2EDuration="2.106602926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:58.105677257 +0000 UTC m=+4626.554764897" watchObservedRunningTime="2026-10-31 01:35:58.106602926 +0000 UTC m=+4626.555690565" Oct 31 01:35:59.065866 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:59.065826 5011 generic.go:356] "Generic (PLEG): container finished" podID="53f8616e-d414-4f94-ace1-08db66347203" containerID="1babd637282a001e02b7d3601b9d7fdd6a4a01b7c8781a33290c05921f2eb0d0" exitCode=0 Oct 31 01:35:59.066296 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:35:59.065912 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" event={"ID":"53f8616e-d414-4f94-ace1-08db66347203","Type":"ContainerDied","Data":"1babd637282a001e02b7d3601b9d7fdd6a4a01b7c8781a33290c05921f2eb0d0"} Oct 31 01:36:00.212233 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.212183 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:36:00.337886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.337802 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\") pod \"53f8616e-d414-4f94-ace1-08db66347203\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " Oct 31 01:36:00.337886 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.337856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\" (UniqueName: \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\") pod \"53f8616e-d414-4f94-ace1-08db66347203\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " Oct 31 01:36:00.338125 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.337981 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\") pod \"53f8616e-d414-4f94-ace1-08db66347203\" (UID: \"53f8616e-d414-4f94-ace1-08db66347203\") " Oct 31 01:36:00.338391 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.338364 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca" pod "53f8616e-d414-4f94-ace1-08db66347203" (UID: "53f8616e-d414-4f94-ace1-08db66347203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:36:00.340024 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.339999 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7" pod "53f8616e-d414-4f94-ace1-08db66347203" (UID: "53f8616e-d414-4f94-ace1-08db66347203"). InnerVolumeSpecName "kube-api-access-s5hs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:00.340133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.339999 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca" pod "53f8616e-d414-4f94-ace1-08db66347203" (UID: "53f8616e-d414-4f94-ace1-08db66347203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:00.439637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.439601 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8616e-d414-4f94-ace1-08db66347203-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:36:00.439637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.439632 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8616e-d414-4f94-ace1-08db66347203-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:36:00.439637 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:00.439642 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5hs7\" (UniqueName: \"kubernetes.io/projected/53f8616e-d414-4f94-ace1-08db66347203-kube-api-access-s5hs7\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:36:01.079656 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:01.079620 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" Oct 31 01:36:01.079823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:01.079649 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7" event={"ID":"53f8616e-d414-4f94-ace1-08db66347203","Type":"ContainerDied","Data":"a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b"} Oct 31 01:36:01.079823 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:01.079708 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e6795060e70a6e7b81defe47612f47c18db7a1cb3b0aa6600116890847676b" Oct 31 01:36:01.946311 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:36:01.946277 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:36:08.836124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:08.836022 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7"] Oct 31 01:36:08.847666 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:08.847635 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e17bf-b7mn7"] Oct 31 01:36:09.681905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:09.681860 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f8616e-d414-4f94-ace1-08db66347203" path="/var/lib/kubelet/pods/53f8616e-d414-4f94-ace1-08db66347203/volumes" Oct 31 01:36:10.841286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:10.841186 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:36:23.692442 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:23.692408 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:36:26.350230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:26.349331 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97mst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:36:26.350230 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:26.349430 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97mst" podUID="cebca123-8758-47ed-9812-b22723173e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:36:31.946202 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:36:31.946165 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:36:40.971479 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:40.971442 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:36:56.197462 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:36:56.197428 5011 scope.go:122] "RemoveContainer" containerID="c44016c58128e85936b010fec6cc9d684b5646a8e49bc5767e4ef6c29e39fca8" Oct 31 01:37:00.299383 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:00.299348 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:37:01.030501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:01.030453 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8"] Oct 31 01:37:01.042334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:01.042298 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a92947e-75vb8"] Oct 31 01:37:01.685408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:01.685369 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8480424-a1b8-47cc-83b8-bd4bd55fce0d" path="/var/lib/kubelet/pods/a8480424-a1b8-47cc-83b8-bd4bd55fce0d/volumes" Oct 31 01:37:02.003763 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:37:02.003677 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:37:14.693817 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:14.693776 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:37:22.470952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:22.470911 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:37:30.994999 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:30.994960 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:37:32.014377 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:37:32.014334 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:37:36.574286 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:36.570055 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:37:54.702188 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:37:54.702152 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:38:01.945396 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:38:01.945356 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:38:12.778864 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:12.778822 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:38:22.235957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.235922 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v"] Oct 31 01:38:22.236417 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.236394 5011 cpu_manager.go:414] "containerMap: removing container" podUID="53f8616e-d414-4f94-ace1-08db66347203" containerName="revoke-keys" Oct 31 01:38:22.236580 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.236559 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f8616e-d414-4f94-ace1-08db66347203" containerName="revoke-keys" Oct 31 01:38:22.236698 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.236688 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f8616e-d414-4f94-ace1-08db66347203" containerName="revoke-keys" Oct 31 01:38:22.239918 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.239899 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.258965 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.258942 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v"] Oct 31 01:38:22.268218 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.268185 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:38:22.268471 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.268452 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:38:22.268572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.268488 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:38:22.268615 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.268590 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:38:22.287807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.287783 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.287895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.287828 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.287937 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.287925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.389171 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.389135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.389372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.389198 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.389372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.389297 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.389924 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.389904 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.391567 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.391539 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:22.417497 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:38:22.417471 5011 projected.go:196] Error preparing data for projected volume kube-api-access-7k26f for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 01:38:22.417620 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:38:22.417529 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f podName:2cedba6f-7113-41be-bd86-f5a54287c5e3 nodeName:}" failed. No retries permitted until 2026-10-31 01:38:22.917514951 +0000 UTC m=+4771.366602582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7k26f" (UniqueName: "kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f") pod "maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" (UID: "2cedba6f-7113-41be-bd86-f5a54287c5e3") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 01:38:22.994086 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:22.994050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:23.017680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.017657 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") pod \"maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:23.153055 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.153025 5011 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" secret="" err="secret \"maas-api-cleanup-dockercfg-v4n7v\" not found" Oct 31 01:38:23.153239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.153074 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:23.451162 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.451131 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v"] Oct 31 01:38:23.451671 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:38:23.451644 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cedba6f_7113_41be_bd86_f5a54287c5e3.slice/crio-f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224 WatchSource:0}: Error finding container f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224: Status 404 returned error can't find the container with id f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224 Oct 31 01:38:23.973978 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.973937 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" event={"ID":"2cedba6f-7113-41be-bd86-f5a54287c5e3","Type":"ContainerStarted","Data":"c57b0c4d441cf58fe6b234970fe38d0ccc9f96a02a62d16b1cf9be0c3c4d07b0"} Oct 31 01:38:23.974124 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.973996 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" event={"ID":"2cedba6f-7113-41be-bd86-f5a54287c5e3","Type":"ContainerStarted","Data":"f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224"} Oct 31 01:38:23.974169 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:23.974125 5011 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" secret="" err="secret \"maas-api-cleanup-dockercfg-v4n7v\" not found" Oct 31 01:38:24.015438 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:24.015356 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" podStartSLOduration=2.015344888 podStartE2EDuration="2.015344888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:24.014697957 +0000 UTC m=+4772.463785597" watchObservedRunningTime="2026-10-31 01:38:24.015344888 +0000 UTC m=+4772.464432527" Oct 31 01:38:24.980177 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:24.980138 5011 generic.go:356] "Generic (PLEG): container finished" podID="2cedba6f-7113-41be-bd86-f5a54287c5e3" containerID="c57b0c4d441cf58fe6b234970fe38d0ccc9f96a02a62d16b1cf9be0c3c4d07b0" exitCode=0 Oct 31 01:38:24.980625 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:24.980237 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" event={"ID":"2cedba6f-7113-41be-bd86-f5a54287c5e3","Type":"ContainerDied","Data":"c57b0c4d441cf58fe6b234970fe38d0ccc9f96a02a62d16b1cf9be0c3c4d07b0"} Oct 31 01:38:26.120539 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.120516 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:26.220087 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.220057 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") pod \"2cedba6f-7113-41be-bd86-f5a54287c5e3\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " Oct 31 01:38:26.220530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.220153 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\") pod \"2cedba6f-7113-41be-bd86-f5a54287c5e3\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " Oct 31 01:38:26.220530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.220193 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\") pod \"2cedba6f-7113-41be-bd86-f5a54287c5e3\" (UID: \"2cedba6f-7113-41be-bd86-f5a54287c5e3\") " Oct 31 01:38:26.220645 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.220612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca" pod "2cedba6f-7113-41be-bd86-f5a54287c5e3" (UID: "2cedba6f-7113-41be-bd86-f5a54287c5e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:26.222138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.222112 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f" pod "2cedba6f-7113-41be-bd86-f5a54287c5e3" (UID: "2cedba6f-7113-41be-bd86-f5a54287c5e3"). InnerVolumeSpecName "kube-api-access-7k26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:26.222138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.222128 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca" pod "2cedba6f-7113-41be-bd86-f5a54287c5e3" (UID: "2cedba6f-7113-41be-bd86-f5a54287c5e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:26.321421 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.321400 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k26f\" (UniqueName: \"kubernetes.io/projected/2cedba6f-7113-41be-bd86-f5a54287c5e3-kube-api-access-7k26f\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:38:26.321543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.321423 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:38:26.321543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.321432 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cedba6f-7113-41be-bd86-f5a54287c5e3-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:38:26.991501 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.991470 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" Oct 31 01:38:26.991781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.991492 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v" event={"ID":"2cedba6f-7113-41be-bd86-f5a54287c5e3","Type":"ContainerDied","Data":"f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224"} Oct 31 01:38:26.991905 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:26.991801 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f08e3c1785468a91e1a4e77fefe7397e2aa8e78eadcab1a73de3376a8f6f4224" Oct 31 01:38:31.947542 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:38:31.947507 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:38:35.390334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:35.390300 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v"] Oct 31 01:38:35.392113 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:35.392082 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c206c91-9hv7v"] Oct 31 01:38:35.680908 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:35.680832 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cedba6f-7113-41be-bd86-f5a54287c5e3" path="/var/lib/kubelet/pods/2cedba6f-7113-41be-bd86-f5a54287c5e3/volumes" Oct 31 01:38:37.026657 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:37.026624 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:38:49.128691 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:49.128655 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:38:52.851788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:52.851760 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:38:52.871700 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:52.871673 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:38:56.227474 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:56.227439 5011 scope.go:122] "RemoveContainer" containerID="a6b13892857df859f894bb885383094996bf48080215e3aade3191aa1d26e400" Oct 31 01:38:56.236776 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:38:56.236746 5011 scope.go:122] "RemoveContainer" containerID="e7a5d5915badf3cefce1439d385a9d5bc8e65ddc7032c9de3bfe58a5192802e9" Oct 31 01:39:01.947911 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:39:01.947867 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:39:31.945618 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:39:31.945580 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:40:01.945875 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:40:01.945836 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:40:19.329265 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.329230 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9"] Oct 31 01:40:19.329676 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.329664 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2cedba6f-7113-41be-bd86-f5a54287c5e3" containerName="revoke-keys" Oct 31 01:40:19.329726 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.329678 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cedba6f-7113-41be-bd86-f5a54287c5e3" containerName="revoke-keys" Oct 31 01:40:19.329769 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.329738 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cedba6f-7113-41be-bd86-f5a54287c5e3" containerName="revoke-keys" Oct 31 01:40:19.332879 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.332862 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.350707 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.350684 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9"] Oct 31 01:40:19.361492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.361468 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:40:19.361612 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.361494 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:40:19.361931 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.361910 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:40:19.362036 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.361980 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmh5j\"" Oct 31 01:40:19.409694 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.409670 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt42v\" (UniqueName: \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.409794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.409702 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.409794 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.409741 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.511017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.510986 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.511138 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.511053 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.511250 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.511159 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt42v\" (UniqueName: \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.511706 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.511686 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.513343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.513327 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.535238 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.535196 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt42v\" (UniqueName: \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.645543 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.645484 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:19.945834 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:40:19.945802 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8191b83_4a85_46a7_b948_700ab67e1fa0.slice/crio-23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e WatchSource:0}: Error finding container 23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e: Status 404 returned error can't find the container with id 23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e Oct 31 01:40:19.945970 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:19.945901 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9"] Oct 31 01:40:20.601201 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:20.601161 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" event={"ID":"e8191b83-4a85-46a7-b948-700ab67e1fa0","Type":"ContainerStarted","Data":"039fd3f592487ca9e66a50e7fabb7fdeb4e7db8584556940ab9511c31a7be121"} Oct 31 01:40:20.601652 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:20.601247 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" event={"ID":"e8191b83-4a85-46a7-b948-700ab67e1fa0","Type":"ContainerStarted","Data":"23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e"} Oct 31 01:40:20.642534 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:20.642487 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" podStartSLOduration=1.6424730300000001 podStartE2EDuration="1.64247303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:20.642129033 +0000 UTC m=+4889.091216674" watchObservedRunningTime="2026-10-31 01:40:20.64247303 +0000 UTC m=+4889.091560669" Oct 31 01:40:21.607974 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:21.607933 5011 generic.go:356] "Generic (PLEG): container finished" podID="e8191b83-4a85-46a7-b948-700ab67e1fa0" containerID="039fd3f592487ca9e66a50e7fabb7fdeb4e7db8584556940ab9511c31a7be121" exitCode=0 Oct 31 01:40:21.608402 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:21.608020 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" event={"ID":"e8191b83-4a85-46a7-b948-700ab67e1fa0","Type":"ContainerDied","Data":"039fd3f592487ca9e66a50e7fabb7fdeb4e7db8584556940ab9511c31a7be121"} Oct 31 01:40:22.753803 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.753780 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:22.839538 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.839506 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\" (UniqueName: \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\") pod \"e8191b83-4a85-46a7-b948-700ab67e1fa0\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " Oct 31 01:40:22.839719 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.839547 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\") pod \"e8191b83-4a85-46a7-b948-700ab67e1fa0\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " Oct 31 01:40:22.839719 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.839599 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\") pod \"e8191b83-4a85-46a7-b948-700ab67e1fa0\" (UID: \"e8191b83-4a85-46a7-b948-700ab67e1fa0\") " Oct 31 01:40:22.839960 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.839935 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca" pod "e8191b83-4a85-46a7-b948-700ab67e1fa0" (UID: "e8191b83-4a85-46a7-b948-700ab67e1fa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:22.841636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.841614 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v" pod "e8191b83-4a85-46a7-b948-700ab67e1fa0" (UID: "e8191b83-4a85-46a7-b948-700ab67e1fa0"). InnerVolumeSpecName "kube-api-access-qt42v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:22.841731 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.841622 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca" pod "e8191b83-4a85-46a7-b948-700ab67e1fa0" (UID: "e8191b83-4a85-46a7-b948-700ab67e1fa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:22.941292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.941201 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt42v\" (UniqueName: \"kubernetes.io/projected/e8191b83-4a85-46a7-b948-700ab67e1fa0-kube-api-access-qt42v\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:40:22.941292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.941248 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:40:22.941292 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:22.941263 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8191b83-4a85-46a7-b948-700ab67e1fa0-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:40:23.619629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:23.619594 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" event={"ID":"e8191b83-4a85-46a7-b948-700ab67e1fa0","Type":"ContainerDied","Data":"23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e"} Oct 31 01:40:23.619629 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:23.619621 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9" Oct 31 01:40:23.619845 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:23.619643 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23415da32ee892f6f8df0263e73fd67ef4113f3021d8bbf73226459e304b3d5e" Oct 31 01:40:31.954360 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:40:31.954316 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:40:36.370632 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:36.370546 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9"] Oct 31 01:40:36.382142 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:36.382105 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-aebdfd-rgjv9"] Oct 31 01:40:37.681986 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:37.681948 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8191b83-4a85-46a7-b948-700ab67e1fa0" path="/var/lib/kubelet/pods/e8191b83-4a85-46a7-b948-700ab67e1fa0/volumes" Oct 31 01:40:37.992792 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:37.992703 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:40:46.078996 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:46.078961 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:40:50.031261 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:50.031202 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn"] Oct 31 01:40:50.043791 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:50.043761 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138f35-xqlhn"] Oct 31 01:40:51.681874 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:51.681840 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395fd545-c908-4206-a7cd-540ca932a30c" path="/var/lib/kubelet/pods/395fd545-c908-4206-a7cd-540ca932a30c/volumes" Oct 31 01:40:54.305372 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:40:54.305337 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:41:00.978586 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:00.978546 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc"] Oct 31 01:41:00.979001 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:00.978985 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e8191b83-4a85-46a7-b948-700ab67e1fa0" containerName="revoke-keys" Oct 31 01:41:00.979052 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:00.979002 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8191b83-4a85-46a7-b948-700ab67e1fa0" containerName="revoke-keys" Oct 31 01:41:00.979090 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:00.979080 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8191b83-4a85-46a7-b948-700ab67e1fa0" containerName="revoke-keys" Oct 31 01:41:00.983569 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:00.983552 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.002119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.002090 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc"] Oct 31 01:41:01.010953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.010928 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:41:01.011119 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.011102 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:41:01.011202 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.011181 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmh5j\"" Oct 31 01:41:01.011268 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.011181 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:41:01.040358 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.040330 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcqz\" (UniqueName: \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.040480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.040375 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.040480 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.040401 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.141011 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.140980 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.141112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.141065 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcqz\" (UniqueName: \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.141112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.141102 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.141766 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.141741 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.143348 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.143331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.164687 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.164661 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcqz\" (UniqueName: \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\") pod \"maas-api-revoke-sub-f03629db-zztlc\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.296463 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.296402 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:01.596406 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:41:01.596366 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fefe5a6_cb11_4a2d_b1e4_a70e8363b6b5.slice/crio-caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f WatchSource:0}: Error finding container caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f: Status 404 returned error can't find the container with id caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f Oct 31 01:41:01.596811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.596783 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc"] Oct 31 01:41:01.888012 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:01.887942 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" event={"ID":"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5","Type":"ContainerStarted","Data":"caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f"} Oct 31 01:41:01.947496 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:41:01.947465 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:41:02.896386 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:02.896351 5011 generic.go:356] "Generic (PLEG): container finished" podID="1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" containerID="e618b3ac4f6e416cdd3427b5ed652008cb444de4c90fda8fa4330a577c3e0650" exitCode=0 Oct 31 01:41:02.896780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:02.896397 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" event={"ID":"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5","Type":"ContainerDied","Data":"e618b3ac4f6e416cdd3427b5ed652008cb444de4c90fda8fa4330a577c3e0650"} Oct 31 01:41:04.044548 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.044523 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:04.178408 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.178334 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\") pod \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " Oct 31 01:41:04.178555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.178428 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\" (UniqueName: \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\") pod \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " Oct 31 01:41:04.178555 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.178459 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\") pod \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\" (UID: \"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5\") " Oct 31 01:41:04.178868 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.178842 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca" pod "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" (UID: "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:41:04.180535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.180504 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca" pod "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" (UID: "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:41:04.180642 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.180512 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz" pod "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" (UID: "1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5"). InnerVolumeSpecName "kube-api-access-xmcqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:41:04.279514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.279490 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:04.279514 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.279514 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcqz\" (UniqueName: \"kubernetes.io/projected/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-kube-api-access-xmcqz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:04.279662 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.279524 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:04.908724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.908694 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" Oct 31 01:41:04.908724 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.908704 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zztlc" event={"ID":"1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5","Type":"ContainerDied","Data":"caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f"} Oct 31 01:41:04.908944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:04.908755 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caac66332ff25eab10a5909f815e86836223e48db7658573b992ab9c6bea065f" Oct 31 01:41:12.309460 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:12.309425 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:41:18.221419 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.221383 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn"] Oct 31 01:41:18.221811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.221790 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" containerName="revoke-keys" Oct 31 01:41:18.221811 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.221802 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" containerName="revoke-keys" Oct 31 01:41:18.221891 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.221878 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5" containerName="revoke-keys" Oct 31 01:41:18.225148 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.225130 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.245380 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.245353 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn"] Oct 31 01:41:18.254812 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.254781 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:41:18.254947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.254808 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:41:18.254947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.254807 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:41:18.254947 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.254819 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmh5j\"" Oct 31 01:41:18.319003 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.318967 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.319159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.319030 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8bz\" (UniqueName: \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.319159 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.319066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.419988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.419955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.420112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.420017 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.420112 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.420083 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8bz\" (UniqueName: \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.420788 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.420761 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.422334 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.422312 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.444927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.444897 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8bz\" (UniqueName: \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\") pod \"maas-api-revoke-sub-0165d7d2-hl2mn\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.538146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.538094 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:18.865917 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:18.865885 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn"] Oct 31 01:41:18.866669 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:41:18.866631 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d35f996_a39b_4b42_96d5_9c13e50aea71.slice/crio-54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911 WatchSource:0}: Error finding container 54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911: Status 404 returned error can't find the container with id 54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911 Oct 31 01:41:19.022988 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:19.022958 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" event={"ID":"1d35f996-a39b-4b42-96d5-9c13e50aea71","Type":"ContainerStarted","Data":"54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911"} Oct 31 01:41:20.029394 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:20.029313 5011 generic.go:356] "Generic (PLEG): container finished" podID="1d35f996-a39b-4b42-96d5-9c13e50aea71" containerID="350613c18b8160f55bd5a9321d6ae588bcbfaa01d6e2abc3d2eae12a07cdd82a" exitCode=0 Oct 31 01:41:20.029753 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:20.029408 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" event={"ID":"1d35f996-a39b-4b42-96d5-9c13e50aea71","Type":"ContainerDied","Data":"350613c18b8160f55bd5a9321d6ae588bcbfaa01d6e2abc3d2eae12a07cdd82a"} Oct 31 01:41:21.169912 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.169884 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:21.245032 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.244992 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\") pod \"1d35f996-a39b-4b42-96d5-9c13e50aea71\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " Oct 31 01:41:21.245179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.245080 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\") pod \"1d35f996-a39b-4b42-96d5-9c13e50aea71\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " Oct 31 01:41:21.245179 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.245133 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\" (UniqueName: \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\") pod \"1d35f996-a39b-4b42-96d5-9c13e50aea71\" (UID: \"1d35f996-a39b-4b42-96d5-9c13e50aea71\") " Oct 31 01:41:21.245365 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.245341 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca" pod "1d35f996-a39b-4b42-96d5-9c13e50aea71" (UID: "1d35f996-a39b-4b42-96d5-9c13e50aea71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:41:21.245535 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.245513 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:21.247099 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.247075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca" pod "1d35f996-a39b-4b42-96d5-9c13e50aea71" (UID: "1d35f996-a39b-4b42-96d5-9c13e50aea71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:41:21.247254 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.247236 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz" pod "1d35f996-a39b-4b42-96d5-9c13e50aea71" (UID: "1d35f996-a39b-4b42-96d5-9c13e50aea71"). InnerVolumeSpecName "kube-api-access-4q8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:41:21.346105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.346074 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8bz\" (UniqueName: \"kubernetes.io/projected/1d35f996-a39b-4b42-96d5-9c13e50aea71-kube-api-access-4q8bz\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:21.346105 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:21.346102 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d35f996-a39b-4b42-96d5-9c13e50aea71-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:22.041129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:22.041055 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" Oct 31 01:41:22.041129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:22.041069 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hl2mn" event={"ID":"1d35f996-a39b-4b42-96d5-9c13e50aea71","Type":"ContainerDied","Data":"54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911"} Oct 31 01:41:22.041129 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:22.041118 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d7550bbe3e05f1b6b2e6e0b67bbbf4a59677b81a32fa1a99085dae898d9911" Oct 31 01:41:29.228895 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.228853 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65"] Oct 31 01:41:29.229530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.229319 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1d35f996-a39b-4b42-96d5-9c13e50aea71" containerName="revoke-keys" Oct 31 01:41:29.229530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.229333 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d35f996-a39b-4b42-96d5-9c13e50aea71" containerName="revoke-keys" Oct 31 01:41:29.229530 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.229412 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d35f996-a39b-4b42-96d5-9c13e50aea71" containerName="revoke-keys" Oct 31 01:41:29.232902 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.232874 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.252877 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.252850 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:41:29.265044 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.265014 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmh5j\"" Oct 31 01:41:29.265266 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.265115 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:41:29.265403 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.265135 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:41:29.267890 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.267862 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65"] Oct 31 01:41:29.313253 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.313194 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvw7s\" (UniqueName: \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.313420 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.313367 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.313492 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.313473 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.414410 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.414377 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvw7s\" (UniqueName: \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.414577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.414443 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.414577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.414479 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.415088 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.415061 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.416771 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.416749 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.438017 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.437982 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvw7s\" (UniqueName: \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\") pod \"maas-api-revoke-sub-6b19338f-m2l65\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.550146 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.550072 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:29.852226 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:29.852174 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65"] Oct 31 01:41:29.852448 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:41:29.852422 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39e6fe1_eeb1_47c1_bf24_4e44c08ad454.slice/crio-2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333 WatchSource:0}: Error finding container 2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333: Status 404 returned error can't find the container with id 2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333 Oct 31 01:41:30.082883 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:30.082847 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" event={"ID":"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454","Type":"ContainerStarted","Data":"2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333"} Oct 31 01:41:31.090366 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:31.090335 5011 generic.go:356] "Generic (PLEG): container finished" podID="b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" containerID="31e001aff1e1e765071525cda9eb2e7d00221dc22e26657e2fbd2e048018cbbc" exitCode=0 Oct 31 01:41:31.090767 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:31.090427 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" event={"ID":"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454","Type":"ContainerDied","Data":"31e001aff1e1e765071525cda9eb2e7d00221dc22e26657e2fbd2e048018cbbc"} Oct 31 01:41:31.945178 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:41:31.945139 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:41:32.229397 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.229374 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:32.343284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.343255 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\") pod \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " Oct 31 01:41:32.343464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.343304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\" (UniqueName: \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\") pod \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " Oct 31 01:41:32.343464 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.343369 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\") pod \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\" (UID: \"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454\") " Oct 31 01:41:32.343712 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.343683 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca" pod "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" (UID: "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:41:32.345564 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.345532 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca" pod "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" (UID: "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:41:32.345680 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.345606 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s" pod "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" (UID: "b39e6fe1-eeb1-47c1-bf24-4e44c08ad454"). InnerVolumeSpecName "kube-api-access-lvw7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:41:32.444729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.444701 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-xks-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:32.444729 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.444728 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-maas-api-openshift-ca\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:32.444875 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:32.444741 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvw7s\" (UniqueName: \"kubernetes.io/projected/b39e6fe1-eeb1-47c1-bf24-4e44c08ad454-kube-api-access-lvw7s\") on node \"ip-10-0-132-66.ec2.internal\" DevicePath \"\"" Oct 31 01:41:33.103251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:33.103200 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" Oct 31 01:41:33.103251 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:33.103235 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-m2l65" event={"ID":"b39e6fe1-eeb1-47c1-bf24-4e44c08ad454","Type":"ContainerDied","Data":"2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333"} Oct 31 01:41:33.103503 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:33.103287 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2154ea5b8662eaee11fbcf9343cc9a3b4f4354ca3770325691c905e14d3c1333" Oct 31 01:41:40.520185 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:40.520153 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8796"] Oct 31 01:41:52.255172 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:52.255140 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fp76r_3f82f539-7ac1-438f-a03d-d48c6f0f28f6/manager/0.log" Oct 31 01:41:52.707276 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:52.707241 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9zr4p_1ac10192-0582-4980-ae86-03b73eb7ce1e/manager/0.log" Oct 31 01:41:53.276340 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:53.276306 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lbvwx_ee72f98b-c3db-47cd-b2ad-492dc4f77bd4/server/0.log" Oct 31 01:41:53.470168 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:53.470140 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cz7rx_f738e6e1-0018-4c35-a89d-6ba0ca8adf0d/manager/0.log" Oct 31 01:41:54.878927 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:54.878893 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hl2mn_1d35f996-a39b-4b42-96d5-9c13e50aea71/revoke-keys/0.log" Oct 31 01:41:55.636423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:55.636384 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-m2l65_b39e6fe1-eeb1-47c1-bf24-4e44c08ad454/revoke-keys/0.log" Oct 31 01:41:55.819772 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:55.819740 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zztlc_1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5/revoke-keys/0.log" Oct 31 01:41:56.293450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:56.293419 5011 scope.go:122] "RemoveContainer" containerID="ec0d25e9cfe6c177bc631fc50ed11566b01500873e97e8909327251d5e013583" Oct 31 01:41:57.795995 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:57.795968 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qfvp6_cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1/manager/0.log" Oct 31 01:41:57.999239 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:57.999190 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bfgql_a139a505-279a-4671-a5f0-62c621eb1b38/manager/0.log" Oct 31 01:41:58.403354 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:58.403322 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nnkvr_590fb320-40cf-438a-a1d3-7d718bb35d18/manager/0.log" Oct 31 01:41:58.719859 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:58.719789 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8796_dbda9462-c01f-4d5c-81c9-a0546b438abb/limitador/0.log" Oct 31 01:41:58.922450 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:58.922409 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-k85h8_42a1cede-f27c-49c3-ab16-e91935a341ab/manager/0.log" Oct 31 01:41:59.699345 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:41:59.699317 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd55n5ms_29ad5251-bc5f-4e4d-8163-11929c64c4aa/istio-proxy/0.log" Oct 31 01:42:00.679598 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:00.679571 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8gm8p_762c1b0e-8055-4782-b0d9-7ca81f06332b/istio-proxy/0.log" Oct 31 01:42:00.864455 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:00.864418 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-97mst_cebca123-8758-47ed-9812-b22723173e81/istio-proxy/1.log" Oct 31 01:42:01.945713 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:42:01.945677 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:42:02.066481 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:02.066457 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zhprr_008798e7-1bb6-4855-ba26-7f3a7dbc89ec/istio-proxy/0.log" Oct 31 01:42:02.487187 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:02.487144 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9_26c6e1ee-8686-46d9-b538-ac2c575ea5f9/istio-proxy/0.log" Oct 31 01:42:03.060463 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:03.060389 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f45b7759-45twl_cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/router/0.log" Oct 31 01:42:03.528344 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:03.528313 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-97mst_cebca123-8758-47ed-9812-b22723173e81/istio-proxy/0.log" Oct 31 01:42:16.005743 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:16.005712 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fp76r_3f82f539-7ac1-438f-a03d-d48c6f0f28f6/manager/0.log" Oct 31 01:42:16.444225 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:16.444190 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9zr4p_1ac10192-0582-4980-ae86-03b73eb7ce1e/manager/0.log" Oct 31 01:42:16.924333 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:16.924306 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lbvwx_ee72f98b-c3db-47cd-b2ad-492dc4f77bd4/server/0.log" Oct 31 01:42:17.123152 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:17.123126 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cz7rx_f738e6e1-0018-4c35-a89d-6ba0ca8adf0d/manager/0.log" Oct 31 01:42:18.338167 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:18.338136 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hl2mn_1d35f996-a39b-4b42-96d5-9c13e50aea71/revoke-keys/0.log" Oct 31 01:42:18.912364 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:18.912336 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-m2l65_b39e6fe1-eeb1-47c1-bf24-4e44c08ad454/revoke-keys/0.log" Oct 31 01:42:19.102667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:19.102633 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zztlc_1fefe5a6-cb11-4a2d-b1e4-a70e8363b6b5/revoke-keys/0.log" Oct 31 01:42:20.670376 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:20.670343 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qfvp6_cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1/manager/0.log" Oct 31 01:42:20.862807 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:20.862778 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bfgql_a139a505-279a-4671-a5f0-62c621eb1b38/manager/0.log" Oct 31 01:42:21.255944 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:21.255914 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nnkvr_590fb320-40cf-438a-a1d3-7d718bb35d18/manager/0.log" Oct 31 01:42:21.485284 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:21.485249 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8796_dbda9462-c01f-4d5c-81c9-a0546b438abb/limitador/0.log" Oct 31 01:42:21.686405 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:21.686380 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-k85h8_42a1cede-f27c-49c3-ab16-e91935a341ab/manager/0.log" Oct 31 01:42:22.264423 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:22.264392 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd55n5ms_29ad5251-bc5f-4e4d-8163-11929c64c4aa/istio-proxy/0.log" Oct 31 01:42:23.280155 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:23.280124 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8gm8p_762c1b0e-8055-4782-b0d9-7ca81f06332b/istio-proxy/0.log" Oct 31 01:42:23.465862 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:23.465834 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-97mst_cebca123-8758-47ed-9812-b22723173e81/istio-proxy/1.log" Oct 31 01:42:24.668595 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:24.668544 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zhprr_008798e7-1bb6-4855-ba26-7f3a7dbc89ec/istio-proxy/0.log" Oct 31 01:42:25.057400 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:25.057323 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9_26c6e1ee-8686-46d9-b538-ac2c575ea5f9/istio-proxy/0.log" Oct 31 01:42:25.603802 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:25.603762 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f45b7759-45twl_cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/router/0.log" Oct 31 01:42:31.945859 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:42:31.945826 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:42:56.313678 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:42:56.313638 5011 scope.go:122] "RemoveContainer" containerID="1babd637282a001e02b7d3601b9d7fdd6a4a01b7c8781a33290c05921f2eb0d0" Oct 31 01:43:01.946541 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:43:01.946494 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:43:31.946763 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:43:31.946726 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:43:49.707781 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:49.707701 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-77tnr_5ef4d6fd-df80-4242-9971-19f9ff9316c5/global-pull-secret-syncer/0.log" Oct 31 01:43:50.262750 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:50.262721 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kw88c_193040bf-22d1-4852-acfe-b2b3dd70b1a8/connection-checker/0.log" Oct 31 01:43:50.380547 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:50.380518 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-msmzl_9b22cf76-b18b-4ce7-84e3-0c4940594b5c/connection-checker/0.log" Oct 31 01:43:50.506133 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:50.506111 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-q7dwz_d7dc021f-37cc-43d4-a4e2-50e449ff0032/connection-checker/0.log" Oct 31 01:43:51.029940 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:51.029908 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-klndj_45b345b2-4e5d-4510-8d14-17dacce01a56/konnectivity-agent/0.log" Oct 31 01:43:51.451951 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:51.451922 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-66.ec2.internal_4683e0633b522c7ed8c0620830924e81/haproxy/0.log" Oct 31 01:43:52.887667 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:52.887639 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:43:52.908818 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:52.908791 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:43:59.466759 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:59.466728 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qfvp6_cf2ea1e0-8b4a-429d-8d74-5fbdb32199d1/manager/0.log" Oct 31 01:43:59.594176 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:59.594151 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bfgql_a139a505-279a-4671-a5f0-62c621eb1b38/manager/0.log" Oct 31 01:43:59.893780 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:43:59.893749 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nnkvr_590fb320-40cf-438a-a1d3-7d718bb35d18/manager/0.log" Oct 31 01:44:00.000953 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.000921 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8796_dbda9462-c01f-4d5c-81c9-a0546b438abb/limitador/0.log" Oct 31 01:44:00.162154 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.162080 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-k85h8_42a1cede-f27c-49c3-ab16-e91935a341ab/manager/0.log" Oct 31 01:44:00.209574 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.209547 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn"] Oct 31 01:44:00.209975 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.209961 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" containerName="revoke-keys" Oct 31 01:44:00.210020 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.209976 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" containerName="revoke-keys" Oct 31 01:44:00.210058 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.210042 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39e6fe1-eeb1-47c1-bf24-4e44c08ad454" containerName="revoke-keys" Oct 31 01:44:00.213132 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.213116 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.234674 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.234649 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn"] Oct 31 01:44:00.327343 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.327321 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"openshift-service-ca.crt\"" Oct 31 01:44:00.327499 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.327480 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"kube-root-ca.crt\"" Oct 31 01:44:00.327966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.327953 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cckr\"/\"default-dockercfg-l8p77\"" Oct 31 01:44:00.354356 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.354333 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-proc\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.354449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.354376 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-lib-modules\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.354449 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.354404 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r67\" (UniqueName: \"kubernetes.io/projected/53525204-4a43-4fac-8de0-c54b596b89bf-kube-api-access-c7r67\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.354571 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.354529 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-podres\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.354636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.354567 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-sys\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455572 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455513 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-proc\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455583 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-lib-modules\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-proc\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455685 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455630 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r67\" (UniqueName: \"kubernetes.io/projected/53525204-4a43-4fac-8de0-c54b596b89bf-kube-api-access-c7r67\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455702 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-podres\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455722 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-lib-modules\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-sys\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455768 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-sys\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.455841 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.455805 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53525204-4a43-4fac-8de0-c54b596b89bf-podres\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.479257 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.479232 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r67\" (UniqueName: \"kubernetes.io/projected/53525204-4a43-4fac-8de0-c54b596b89bf-kube-api-access-c7r67\") pod \"perf-node-gather-daemonset-pccmn\" (UID: \"53525204-4a43-4fac-8de0-c54b596b89bf\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.525901 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.525876 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:00.841599 ip-10-0-132-66 kubenswrapper[5011]: W1031 01:44:00.841566 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod53525204_4a43_4fac_8de0_c54b596b89bf.slice/crio-6220b675ab1a7befc3eefd6a9e7d3afc5be07858da359d3a3bf92f5bed40891f WatchSource:0}: Error finding container 6220b675ab1a7befc3eefd6a9e7d3afc5be07858da359d3a3bf92f5bed40891f: Status 404 returned error can't find the container with id 6220b675ab1a7befc3eefd6a9e7d3afc5be07858da359d3a3bf92f5bed40891f Oct 31 01:44:00.841949 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.841891 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn"] Oct 31 01:44:00.911819 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:00.911792 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" event={"ID":"53525204-4a43-4fac-8de0-c54b596b89bf","Type":"ContainerStarted","Data":"6220b675ab1a7befc3eefd6a9e7d3afc5be07858da359d3a3bf92f5bed40891f"} Oct 31 01:44:01.918429 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:01.918395 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" event={"ID":"53525204-4a43-4fac-8de0-c54b596b89bf","Type":"ContainerStarted","Data":"26d7661078d134c10b989ac8a4e0a2fcdc168e81a3e7d98f435ab7789e478583"} Oct 31 01:44:01.918860 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:01.918449 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:01.945037 ip-10-0-132-66 kubenswrapper[5011]: E1031 01:44:01.945007 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qxclf" containerName="extractor" Oct 31 01:44:01.959984 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:01.959935 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" podStartSLOduration=1.959921386 podStartE2EDuration="1.959921386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:01.958525527 +0000 UTC m=+5110.407613165" watchObservedRunningTime="2026-10-31 01:44:01.959921386 +0000 UTC m=+5110.409009024" Oct 31 01:44:02.370277 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:02.370243 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-slzmm_3b8ed8c6-29da-49fc-962a-a0f2bc51f12a/cluster-monitoring-operator/0.log" Oct 31 01:44:03.123617 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:03.123585 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lpkq_f24d7ea1-830e-4d93-abb1-21aee7bea13d/node-exporter/0.log" Oct 31 01:44:03.250693 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:03.250664 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lpkq_f24d7ea1-830e-4d93-abb1-21aee7bea13d/kube-rbac-proxy/0.log" Oct 31 01:44:03.367151 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:03.367127 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lpkq_f24d7ea1-830e-4d93-abb1-21aee7bea13d/init-textfile/0.log" Oct 31 01:44:07.933281 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:07.933253 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-pccmn" Oct 31 01:44:08.393326 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:08.393295 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-pxts4_f0db8503-b76b-46c3-8501-b2a9acedd16a/networking-console-plugin/0.log" Oct 31 01:44:09.455426 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:09.455397 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/2.log" Oct 31 01:44:09.456983 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:09.456963 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qb96d_d34220d9-9132-4909-a542-35d25dd1af45/console-operator/3.log" Oct 31 01:44:11.899636 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:11.899605 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-hrljw_c491ada7-6f46-4e40-97f0-699cc84dc60d/volume-data-source-validator/0.log" Oct 31 01:44:13.411609 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:13.411581 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd55n5ms_29ad5251-bc5f-4e4d-8163-11929c64c4aa/istio-proxy/0.log" Oct 31 01:44:14.326577 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:14.326542 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d476j_10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b/dns/0.log" Oct 31 01:44:14.448966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:14.448943 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d476j_10fec28b-c25d-44f8-a7b2-10e3a4c8cf3b/kube-rbac-proxy/0.log" Oct 31 01:44:15.225042 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:15.225011 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9cp4_e127939f-dcd5-405f-97c6-87b7a68a5c36/dns-node-resolver/0.log" Oct 31 01:44:16.896415 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:16.896386 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-krnph_c5adc70d-63ec-4ab8-9ca5-8fb4928040fc/node-ca/0.log" Oct 31 01:44:18.841023 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:18.840991 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd55n5ms_29ad5251-bc5f-4e4d-8163-11929c64c4aa/istio-proxy/0.log" Oct 31 01:44:18.973957 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:18.973926 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8gm8p_762c1b0e-8055-4782-b0d9-7ca81f06332b/istio-proxy/0.log" Oct 31 01:44:19.042488 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:19.042463 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-97mst_cebca123-8758-47ed-9812-b22723173e81/istio-proxy/0.log" Oct 31 01:44:19.043966 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:19.043950 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-97mst_cebca123-8758-47ed-9812-b22723173e81/istio-proxy/1.log" Oct 31 01:44:19.463336 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:19.463308 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zhprr_008798e7-1bb6-4855-ba26-7f3a7dbc89ec/istio-proxy/0.log" Oct 31 01:44:19.597952 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:19.597917 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-84pz9_26c6e1ee-8686-46d9-b538-ac2c575ea5f9/istio-proxy/0.log" Oct 31 01:44:19.821385 ip-10-0-132-66 kubenswrapper[5011]: I1031 01:44:19.821360 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f45b7759-45twl_cea8d015-5b9b-4bf4-b7b1-cd26b741a77c/router/0.log"